Why "Price Per Token" Doesn't Tell You What a Model Actually Costs

An OpenAI price card says $5 per million input tokens. Anthropic's says $2 per million. Simple enough — Anthropic is cheaper, right? Not necessarily. OpenAI's Tibo Sottiaux, who works on Codex and ChatGPT, posted a comparison on August 16, 2026 that got 673,500+ views making the opposite case: a token from one model isn't the same unit as a token from another, so comparing prices per token across models can be comparing the wrong thing entirely.

This matters directly for anyone choosing between models on explainx.ai or building an agent harness that routes across providers — the sticker price per million tokens is not the same as the price per unit of work.

The pizza analogy

Tibo's framing, in his own words:

"Imagine two identical pizzas. One is cut into 8 slices at $2 each. The other is cut into 16 slices at $1.25 each. The second place advertises cheaper slices, but the whole pizza costs $20 instead of $16. Bummer... your stomach doesn't actually care about the number of slices you just ate."

The 16-slice pizza looks cheaper on the menu board — 37.5% cheaper per slice — and costs 25% more overall, because it takes twice as many slices to eat the same pizza. Tokens work the same way. A model that tokenizes text into more, smaller pieces can charge less per piece and still bill you more for the same document.

What Tibo actually measured

Tibo ran a small comparison spanning English, technical, multilingual, and numerical text through two tokenizers:

That's a 34.5% difference in token count for identical content. If both models charged the exact same price per million tokens, the Opus 5 request would still cost roughly 53% more than the Sol request, purely from tokenizer differences — before either model's actual $/M rate enters the comparison at all.

A follow-up reply from developer Kunal Chaturvedi tested the effect at scale: running 150 Lorem Ipsum paragraphs through 16 different tokenizers pulled from Hugging Face. Using Tibo's own $2/M vs. $1.25/M example prices, the less token-efficient ("red") tokenizer worked out to $0.053 for the batch versus $0.031 for the more efficient ("blue") one — a ~70% higher cost, even though red's headline price per token was only 60% higher than blue's.

Why tokenizers differ at all

A tokenizer is the model-specific system that breaks text into the chunks a model actually processes and bills — see explainx.ai's guide to what LLM tokens are for the mechanics. Different labs train different tokenizers, with different vocabulary sizes and different rules for splitting words, code, whitespace, and non-English text. The practical result:

• English prose tokenizes fairly similarly across most modern tokenizers, but not identically.
• Code, technical notation, and rare identifiers can tokenize very differently — a tokenizer with a smaller vocabulary breaks unfamiliar strings into more, shorter pieces.
• Non-English languages often show the widest gaps, since most tokenizers are still trained on English-majority corpora and encode other languages less efficiently.
• Numbers and structured data (dates, IDs, tables) are a common source of unexpected token bloat, depending on how a tokenizer chunks digit sequences.

This is the same underlying mechanic covered in explainx.ai's full token-pricing breakdown — the $5-input/$30-output sticker price is only one half of the actual cost formula. Tibo's post supplies the other half: the token count itself isn't standardized across models, so the same sticker price can produce very different real-world bills depending on which model is doing the counting.

Most modern tokenizers use some variant of byte-pair encoding (BPE) or a related subword scheme: the training process starts from individual characters or bytes and repeatedly merges the most frequent adjacent pairs into single tokens, building up a fixed vocabulary (commonly somewhere in the tens of thousands to low hundreds of thousands of entries). A larger vocabulary can represent more common words and phrases as single tokens, which tends to make prose more token-efficient — but it also makes the model larger and more expensive to train and serve, so labs make different tradeoffs. A vocabulary trained mostly on English GitHub repos and English web text will represent common English words efficiently and fall back to shorter, more numerous fragments for anything outside that distribution — non-Latin scripts, rare technical terms, unusual variable names, or heavily numeric data. That's the mechanical reason two labs' tokenizers can diverge by 30%+ on the same input: they were built from different training corpora with different vocabulary sizes and different merge rules, not from any shared standard.

A worked example

Take a hypothetical 100,000-token document (measured on Tokenizer A) that a less efficient Tokenizer B renders as 140,000 tokens for the identical text — a 40% gap, in the range Tibo's and Chaturvedi's examples both landed in. At $3 per million input tokens on Model A and $2.20 per million on Model B — Model B looks 27% cheaper on the price card — the real bill comes out to $0.30 for Model A versus $0.308 for Model B. The "cheaper" model is actually slightly more expensive once its tokenizer's real output is priced in, and that gap widens further if Model B's output tends to run more verbose by default, compounding the token-count disadvantage on the pricier output side of the bill.

The reply that mattered: weekly limits are a token story too

Not every reply agreed the pizza math was the whole picture. Developer James Findsout pushed back with a related, practical complaint: "However you try to spin it codex weekly limits have got so much worse... These days the weekly limits last a day or so." That's a useful reminder that token efficiency and usage-limit generosity are two separate levers — a more token-efficient model can still leave you rate-limited faster if the underlying weekly allowance shrinks, a dynamic explainx.ai has tracked in detail on the Claude usage limits timeline and the Codex vs. Claude Code rate-limit comparison. Tokenizer efficiency changes how far your allowance stretches; it doesn't change the size of the allowance itself.

Even a corrected comparison misses the real question

Tibo's own conclusion goes a step further than "adjust for tokenizer differences and you're done":

"Even correcting for tokenizer differences misses the bigger point. What actually matters is price per successful outcome, and for that you can use benchmarks as a starting point, but really you have to try it and measure on your own use cases."

Two models can tokenize identically and still differ wildly on cost per completed task, because:

• One model might need more turns or retries to get a coding task right, multiplying its effective token spend regardless of its tokenizer.
• One model might produce longer, more verbose output by default for the same prompt, independent of tokenizer efficiency.
• Reasoning tokens, when a model uses hidden chain-of-thought that still bills, can dwarf tokenizer-level differences entirely — see the reasoning-token billing section of explainx.ai's pricing guide.

How to actually compare model costs

1. Don't trust the $/M sticker price alone. Treat it as one input, not the answer.
2. Run your own representative text through each model's tokenizer before comparing prices — most providers expose a tokenizer tool or a usage field in the API response that reports the real count.
3. Multiply out the full request, not just input tokens — see the base cost formula for input, output, caching, and reasoning tokens combined.
4. Benchmark on your own use case. Public benchmarks are a starting point, not a substitute — a model that's cheaper on a generic coding benchmark can still be pricier on your specific codebase's token profile.
5. Track cost per completed task, not cost per API call. A model that fails a task and needs a retry effectively doubles its real price regardless of its tokenizer or its sticker rate.

Related on explainx.ai

• GPT-5.6 Sol "50% Off" on OpenRouter: Promo, Not a Price Cut — another sticker-price trap: a router promo badge that isn't a real OpenAI price change
• AI Token Pricing, Explained Without the Pricing-Page Fog — the full input/output/caching cost model
• What Are LLM Tokens? — how tokenizers actually chunk text
• Claude Usage Limits in 2026: Every Change Explained — weekly caps, separate from per-token pricing
• Claude Code vs. Codex: Rate Limit Boost — the usage-allowance side of this same debate
• Anthropic's Sonnet 5 Pricing Made Permanent — a live example of a $/M rate holding steady
• DeepSeek V4 Prices Just Went Up — another current pricing comparison across providers
• Choose Open-Weight vs. Closed AI Models — cost is one factor among several in that decision

Figures reflect Tibo Sottiaux's August 16, 2026 X post and its top replies, as of publication. Tokenizer output for a given text can change if a provider updates its tokenizer, so treat the specific counts above as a snapshot, not a permanent benchmark — measure your own text on the current tokenizer before budgeting.