The Tokenizer Tax: Why Non-English AI Users Pay Up to 17x More Tokens

AI models can charge you more simply because of the language you type in. A study reported by Rest of World on October 26, 2026 says models need far more tokens for some languages than for others, a hidden "tokenizer tax" that raises cost, adds latency and eats context. Among the 20 most popular languages on the internet, the most expensive used between 1.5 and 5 times more tokens than the cheapest. Across 200 languages, the gap in one model grew to 17 times.

If you build or buy AI products for users outside English, this is a line item you may not have budgeted. Below we summarize what the report says, separate what is attributed to the study from what we could confirm independently, and give you a way to measure the tax on your own prompts.

TL;DR: what the report says

What is a token, and why does language change the count?

Language models do not read letters or words. A tokenizer first breaks text into tokens, chunks learned from the data it was built on. If a language appeared often in that data, common words become single tokens. If it appeared rarely, or uses a script the tokenizer saw little of, the same sentence shatters into many small pieces, sometimes down to individual bytes.

The report illustrates this with the sentence "People should benefit from AI, regardless of the language they speak." Using OpenAI's public tokenizer, the English version splits into recognizable words, while the Malayalam version gets "multiple fragments per word without respect for multi-byte characters."

That matters because tokens are the unit everything is measured in: price per million tokens, how fast the answer streams, and how much fits in the context window. Our explainers on what LLM tokens are and AI token pricing cover the basics. The tokenizer tax is what happens when the same meaning costs a different number of tokens depending on language.

Three costs, not one

The study's quoted text frames the problem beyond billing: "Higher token counts translate into higher latency and reduced effective context capacity. These penalties hit hardest where compute and connectivity are already limited."

1. Price. Every prompt and completion is billed per token. The study gives an example: "A startup in [Bengaluru] may pay a surcharge on every prompt and completion while its competitor in Austin does not."
2. Latency. More tokens to generate means a slower answer, since models produce tokens one at a time. Our piece on tokens per second explains why.
3. Context. A fixed window holds less text in a high-token language. Pratyush Choudhury of Activate AI gave Rest of World a tutoring example: if a student's earlier mistakes and explanations eat the window faster in one language, "the disadvantage reaches into the quality of the service." See our guide to context windows and context window pricing.

Script matters as much as how rare the language is

One finding runs against intuition. You might assume the tax simply tracks how many people speak a language or how much of it is online. The report says that is not the whole story. Asturian has relatively few speakers and no Common Crawl data, yet it "benefits from the Latin-script advantage, as well as the representation from other related Romance languages like Italian."

By contrast, languages written in scripts the tokenizer rarely saw were at the greatest disadvantage across nearly all models: Burmese, Tibetan, Tifinagh, Khmer, Lao and Ol Chiki. For builders, the practical lesson is that you cannot predict your cost from a language's speaker count. You need to measure in the actual script your users type.

Is this a new finding?

The idea is not new. Academic work has documented it before, including the 2023 EMNLP paper Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models, which examined OpenAI's API across 22 languages and found that speakers of many supported languages are overcharged while getting poorer results. A 2025 paper, Tokenization Disparities as Infrastructure Bias, looked at more than 200 languages. What Rest of World adds is a current framing tied to startups in India and a call for labs to act.

The earlier work is why the numbers in the report are plausible, but it is not a confirmation of the exact 1.5x to 5x and 17x figures, which Rest of World attributes to the coalition's study.

What we could and could not verify

Honesty about sourcing matters here. We read Rest of World's article in full. We searched for the coalition's own publication and could not find a paper or site under the name "Coalition for Language Equity in AI Research." That does not mean it does not exist, only that we could not open it. Treat the specific figures as reported by Rest of World, not as numbers we reproduced.

What you can do is replicate the effect yourself. The direction of the result, that non-Latin scripts and under-represented languages need more tokens, is consistent with the peer-reviewed 2023 paper and with tokenizer behavior anyone can observe in minutes. The exact multiples depend on the model, the tokenizer version, and the text used.

How to measure the tokenizer tax on your own prompts

Pick 50 to 100 representative sentences from your product, translated by a human into each target language, and count tokens against the model you plan to use. A minimal example with OpenAI's open tiktoken library:

import tiktoken

enc = tiktoken.get_encoding("o200k_base")

samples = {
    "en": "People should benefit from AI, regardless of the language they speak.",
    "hi": "लोगों को एआई से लाभ मिलना चाहिए, चाहे वे कोई भी भाषा बोलें।",
}

base = len(enc.encode(samples["en"]))
for lang, text in samples.items():
    n = len(enc.encode(text))
    print(lang, n, round(n / base, 2))

Swap the encoding for the tokenizer of whichever model you use. Many providers expose a token-counting endpoint, and open models ship their tokenizer files. Then multiply the ratio by your monthly token volume and your price per million tokens to get the surcharge in dollars. Do this for both input and output, since output is often priced higher.

What builders can do about it

• Budget per language, not per request. Add language to your cost dashboard. A flat per-request estimate hides the gap.
• Compare tokenizers, not just prices. Our post on why price per token can mislead shows that a cheaper model with a worse tokenizer can cost more for your workload.
• Test models built for your languages. Indian-language efforts such as Sarvam, BharatGen and NVIDIA's open Indic pretraining pipeline are aimed at this problem, and their tokenizers may split your language more efficiently.
• Consider vocabulary specialization. One write-up we covered, VocabX from Dharma AI, cut inference cost by trimming vocabulary for a narrow task. It is a different problem, but it shows tokenizer choices are a real lever.
• Use prompt caching and trim history. In a high-token language, long conversation history is expensive. Summarize older turns and cache static instructions.

Will falling prices make this go away?

Partly, but not entirely. Our timeline of OpenAI and Anthropic price cuts shows prices per token have dropped sharply this year. Dheemanth Reddy, CEO of Maya Research, told Rest of World the direct cost impact "should become less severe over time," but that "fixing the tokenization gap will still require labs to deliberately design future tokenizers to handle these languages and scripts better."

A price cut shrinks the absolute gap in dollars while leaving the ratio unchanged. A 5x multiplier on a cheaper base is still a 5x multiplier, and the latency and context penalties do not go away with a price cut.

What people are asking labs to do

Choudhury told Rest of World that "language efficiency should be a product requirement, alongside accuracy, speed, and reliability," and that "perfect equality may be unrealistic" but "large, persistent disparities should still require an explanation and an improvement plan." His ask: every major lab should publish a language-by-language scorecard showing what it costs to complete comparable tasks successfully, with measurable improvements for the worst-served languages.

Open data is the other lever. Raoul Nanavati of Navana.ai pointed to RESPIN, a collaboration with the Indian Institute of Science funded by the Gates Foundation, which collected text from native speakers of 38 dialects along with read speech. Better data helps models, but as Reddy warned, better training data alone does not fix the tokenizer.

What this means for what you build or pay

If your users write in Hindi, Tamil, Malayalam, Thai, Burmese or Khmer, assume your effective price per message is higher than the list price suggests, test it, and put the ratio in your pricing model. If you sell to those markets, a model with a better tokenizer for the language can be the cheaper choice even at a higher list price. If you are a lab or tool vendor, publishing per-language token ratios would answer the exact request the report ends on.

Related reading

• What are LLM tokens?
• AI token pricing explained
• Why price per token is misleading
• Context window pricing decoded
• Sarvam AI capabilities and API guide
• BharatGen and India's 22-language sovereign AI

Figures are as reported by Rest of World on October 26, 2026. Tokenizers and prices change often, so re-measure before you budget.