Why Token-Based LLM Price Comparisons Can Be Misleading

Updated

Token-based price comparisons are a useful starting point, but they are not a reliable way to rank models by real cost. The LLM pricing calculator at llm-prices.com states this directly: "Different models use different tokenizers, so direct token-based price comparisons may not be entirely accurate." The core reason is that a per-million-token price only tells you the rate — it does not tell you how many tokens your specific text will consume under each model's tokenizer.

The tokenizer problem

A tokenizer splits text into tokens, and different models use different tokenizers. The same sentence can produce different token counts depending on which model processes it.

For example, a block of English prose, code, or non-Latin script may tokenize more efficiently in one model than another. If Model A charges less per million tokens but needs more tokens to represent your input, the cheaper-looking rate can produce a higher actual bill.

This means a price table sorted by cost per million tokens is comparing rates, not outcomes. The calculator's own note flags this limitation rather than hiding it.

Why per-token prices miss the real cost of a task

Even setting tokenizers aside, per-token pricing does not capture what a task actually costs:

  • Input vs. cached input vs. output are priced differently. A workload heavy on output tokens behaves very differently from one heavy on cached input.
  • Token counts vary with the content. The same instruction in different languages, formats, or lengths will not consume the same number of tokens.
  • Retries and multi-step calls add up. A task that takes several calls costs more than a single-call estimate suggests.
  • Caching changes the math. Cached input tokens are typically cheaper, so how much of your prompt is cacheable matters as much as the base rate.

What the calculator is actually good for

The llm-prices.com calculator is built around three inputs — input tokens, cached input tokens, and output tokens — plus the per-million rates for each. It returns a total cost. That makes it a solid tool for:

  • Estimating the cost of a specific, known workload once you have token counts.
  • Comparing models on the same token assumptions, which isolates the rate difference.
  • Checking how caching shifts the total for a given prompt pattern.

It is not a tool for declaring one model "cheaper" in general, because the token counts themselves are model-dependent.

How to compare more reliably

  1. Measure tokens per model, not per text. Run your actual prompt through each model's tokenizer (or its token-counting endpoint) to get real counts.
  2. Feed those counts into the calculator separately. Use each model's own input, cached input, and output token numbers rather than one shared estimate.
  3. Model the full task. Include expected output length, retries, and how much input can be cached.
  4. Compare totals, not rates. The number that matters is the total cost for your workload, not the headline price per million tokens.

The takeaway

Per-token prices are a rate, and rates only become costs once you know the token counts. Because tokenizers differ across models, the same text can cost different amounts even at identical per-token prices. Use the calculator to turn token counts into totals for a defined workload, and treat any ranking based purely on price-per-million-tokens as an approximation rather than a verdict.