AI & Automation

What a Token Actually Costs and How an Inference Bill Adds Up

Read an inference bill line by line and predict the next one: what a token is, why output costs several times more than input, and which four choices set almost the whole number.

  • Intermediate
  • 12 min total
  • 13 chapters

What decision this helps you make: What one use of your AI feature costs, what it will cost at ten times the volume, and which of the four cost drivers to change first when the bill is too high.

What this topic is

A token is a chunk of text — roughly three quarters of a word in ordinary English — and it is the unit almost every model provider bills in. Two meters run at once: everything you send in, priced one way, and everything the model writes back, priced several times higher. The bill for a feature is therefore not one number but a short multiplication you can do on paper: tokens in, plus tokens out at their own rate, times how many times the feature runs, plus the calls you did not count.

Why it matters

Inference is the first genuinely variable cost most software businesses have ever had. Software margins were built on the fact that the ten-thousandth user cost almost nothing to serve; a feature that spends real money every time someone presses a button breaks that assumption, and it breaks it unevenly — a small minority of users can generate most of the bill. Until you can compute the cost of one use, you cannot price the feature, cannot forecast the bill, and cannot tell whether an expensive customer is a problem or a compliment.

Who should learn it

Operators and founders who have just received a surprising invoice, product managers pricing an AI feature, and anyone deciding whether a workflow is worth automating at the price the arithmetic actually implies.

What you will understand

  • What a token is, how to estimate one from a document you already have, and the four-character rule
  • The full cost stack of a single call — system prompt, retrieved context, history, question, answer
  • Why a long conversation costs far more than the sum of its messages, and the two fixes
  • How to build a monthly forecast from cost per call, and where the forecast usually goes wrong

Prerequisites

Common misconception

"The tokens are cheap, so the bill will be small." Each token is cheap and almost nothing about a real feature involves one call with a short prompt. The bill is set by the multiplication, and the multiplication has terms most people leave out: a system prompt sent on every single call, retrieved documents that dwarf the question, a conversation history re-sent from the beginning every turn, retries after a failure, a validation pass over the answer, and the small fraction of users who use the feature twenty times a day. Nobody is ever surprised by the price per token. They are surprised by how many times it appears.