AI & Automation

Latency, Quality, and Cost: the Three-way Trade in Choosing a Model

Turn "which model should we use?" into a decision with three numbers (a latency budget, a quality floor, and a cost per call), and learn the routing arithmetic that gets you most of the quality at half the price.

  • Advanced
  • 12 min total
  • 14 chapters

What decision this helps you make: Which model handles which request, what to promise about response time, and where a cheaper or faster option is genuinely acceptable rather than merely tempting.

What this topic is

Every model choice trades three things against each other. Bigger models generally answer better, cost more per call and take longer to respond. Smaller ones are fast and cheap and lose accuracy on the hard cases. You cannot maximise all three, so the workable method is to fix two as constraints (a latency budget the surface has to meet and a quality floor measured on your own evaluation set) and then minimise cost subject to both. What makes it interesting is that the constraints differ enormously by surface, so the right answer is usually several models rather than one.

Why it matters

Teams pick one model for the whole product and then argue about it, which guarantees that some surfaces are too slow and others are paying for quality nobody can perceive. Latency is also worse than it looks in production: response times are set by the tail rather than the average, waiting time explodes as capacity fills, and a workflow that makes several calls inherits the slowest one each time. Getting this right is usually worth more than any single model upgrade, and it is entirely within your control.

Who should learn it

Product and engineering leaders choosing models per surface, operators being asked to approve an expensive default, and anyone whose users have started describing the product as slow.

What you will understand

  • What actually makes up a response time, and why the first token and the last one have different causes
  • Why the average latency is the wrong number and the ninety-fifth percentile is the right one
  • The routing and cascade arithmetic: costs, and what each does to the latency distribution
  • How queueing turns a capacity problem into a latency problem long before you run out of capacity

Prerequisites

Common misconception

"Use the best model everywhere: quality is what matters." Quality matters where it is perceptible and where being wrong costs something, and on a large share of production traffic neither condition holds. Meanwhile the best model is usually also the slowest, and slowness is perceptible everywhere: it shows up in abandonment, in complaints, and in the surfaces people quietly stop using. A single default model is a decision to be too slow in the places that need speed and too expensive in the places that need neither. And it is made by not deciding, which is why it survives so long.