AI & Automation

Evaluation Sets and the Moat of Being Able to Prove Your Model Is Right

Build the one AI asset that appreciates instead of depreciating: a labelled set of your customers' own cases that decides what "right" means. Then learn the sample-size arithmetic that tells you whether your last improvement was real.

  • Advanced
  • 14 min total
  • 14 chapters

What decision this helps you make: What to measure, on how many cases, labelled by whom, before you ship a change to a model or a prompt, and whether the improvement you just saw is distinguishable from noise.

What this topic is

An evaluation set is a fixed collection of real inputs paired with the outputs a knowledgeable person agrees are correct, held aside and used to score any version of your system. In machine learning it is the held-out test set. In a business it is something more specific and more valuable: an encoded definition of what your customers consider a correct answer, in their domain, on their documents, including the awkward cases nobody writes down. It is the thing that converts "the new model seems better" into a number with an error bar on it.

Why it matters

When the models themselves stop differentiating, the ability to prove your output is right becomes the differentiator, and unlike model quality, it is an asset you own. It is also the only defence against the two failure modes that make AI products embarrassing: a change that improves the demo and degrades the real work, and a system that has quietly learned the wrong thing and passes every test you thought to write. A competitor can rent your model this afternoon. They cannot rent eighteen months of your customers arguing about which answers were acceptable.

Who should learn it

Anyone shipping AI into a workflow where being wrong has a cost: product and engineering leads, operators buying an AI tool who need to know what to ask for, and founders whose differentiation has to survive the next model release.

What you will understand

  • How large an evaluation set has to be before a three-point improvement is distinguishable from noise
  • Why a random sample cannot measure a rare failure mode, and what to sample instead
  • How to read an evaluation report for the four things that make a headline number meaningless
  • Why an evaluation set is a durable asset while a prompt library and a fine-tune are not

Prerequisites

Common misconception

"We tested it and it got 94%." That sentence has four missing pieces and each of them can reverse the conclusion. On how many cases? 94% of fifty is 47 right and 3 wrong, and the interval around it spans most of the useful range. Selected how? A set that omits the hard cases measures the easy ones. Labelled by whom, and did two people agree? If your own experts disagree on a fifth of the cases, no score above about eighty per cent means anything. And how many times has this set been used to choose between versions? A set you have optimised against twenty times has stopped being a test and become part of the training process.