Model Risk and When a Model Should not Be Allowed to Decide

What this topic is

Model risk is the risk of adverse consequences from decisions based on incorrect or misused model output — a definition supervisors settled on well before the current wave of machine learning, and one that deliberately covers two distinct failures: the model is wrong, and the model is right but used for something it was never built for. Managing it means development discipline, independent validation, and governance that says who may use which model for what.

Why it matters

The question of whether a model decides or advises is usually settled by whoever built it, on the basis of how accurate it is. That is the wrong basis. Accuracy is one input; the others are what a wrong answer costs in each direction, whether it can be reversed, whether you can explain it to the person it affected, and whether a regulator has already made the choice for you. The same fraud model, at the same accuracy, loses money as an autonomous decider and makes a quarter of a million dollars a month as a ranker feeding a queue. Nothing about the model changed.

Who should learn it

Executives approving automated decisions, risk and compliance functions building model governance, and anyone whose model is about to be pointed at customers.

What you will understand

• How to compute the value of a model under three authority levels — decide, review, flag — on the same confusion matrix
• Why precision at realistic base rates is far worse than accuracy figures suggest, and what that does to an automated decision
• The specific failure modes that no accuracy metric detects: drift, feedback loops, selective labels, proxy discrimination, and specification gaming
• A concrete framework for assigning decision authority, and the tripwires that revoke it

Prerequisites

• Synthetic Control and the Constructed Counterfactual
• Correlation, Confounding, and the Causal Question Underneath

Common misconception

"If the model is accurate enough, it should decide." Accuracy is not the variable. A model with 85% recall and a 2% false-positive rate on a 0.4% base rate produces about six false alarms per genuine catch, and whether that is excellent or ruinous depends entirely on what a false alarm costs relative to a catch — and on whether the person on the wrong end of one can get it reversed. Two businesses can rationally give the identical model opposite authority. The decision is about costs, reversibility, and accountability, and accuracy is one term in it rather than the answer.