Quantitative Methods

Base Rates and the Prosecutor's Fallacy in Business Data

Understand why a 98% accurate model can be wrong nine times out of ten, and learn the one calculation that sets the threshold on every flagging system you own.

  • Advanced
  • 10 min total
  • 12 chapters

What decision this helps you make: Where to set the threshold on any screen (fraud, churn, quality, security, hiring), and whether a flag is worth a human being's time at all.

What this topic is

A base rate is how common something is before you look at any evidence. The prosecutor's fallacy is the error of reading the probability of the evidence given a hypothesis as the probability of the hypothesis given the evidence. Together they explain why highly accurate detectors produce mostly false alarms whenever the thing being detected is rare, which, in business, it nearly always is.

Why it matters

Every company now runs models and rules that flag things: fraudulent transactions, accounts about to churn, suspicious expenses, defective units, risky applicants, security events. Each one is a rare-event detector, and each is routinely judged on accuracy, which is the single least informative metric available for a rare event. Getting the base rate arithmetic right determines the size of your review team, the threshold you set, whether an alert reaches a human, and whether the whole system earns its keep.

Who should learn it

Operators who own an alerting or screening process: risk and fraud leads, customer success teams running churn models, quality managers, compliance officers, and anyone about to buy a vendor model on the strength of its accuracy claim.

What you will understand

  • Why precision collapses as the base rate falls, and how to compute it in thirty seconds
  • The odds form of Bayes's rule, which makes the whole calculation a single multiplication
  • How to derive the break-even precision that sets your threshold from cost per review and loss per event
  • How to spot the transposed conditional in the sentence someone is about to persuade you with

Prerequisites

Common misconception

"Our model is 98% accurate, so a flag is very likely to be real." Accuracy is the fraction of ALL cases classified correctly, and when 99.8% of cases are negative, a model that flags nothing at all is 99.8% accurate. The number that matters is precision: of the things you flagged, what share were real. At a 0.2% base rate, a detector with 90% catch rate and 2% false positive rate produces 8,704 flags a month of which 720 are fraud, a precision of 8%. Nothing is wrong with the model. The base rate did that, and no amount of model quality removes it.