explainx / blog
Tokens are the standard units large language models use to read and generate text. Here is what they are, how they differ from words, why input and output are billed separately, and how they connect to context limits, subscriptions, and API pricing—without the jargon pile-on.
Apr 22, 2026
Context length is the cap on 'how much the model can read at once,' not the same as how many parameters it has. This guide defines the window, input vs max output, long-context tradeoffs, and what Anthropic, OpenAI, Google, and Meta publish today.
Apr 22, 2026
Bigger is not a synonym for smarter, but parameter count is still the first axis people use to compare scale. This guide explains what parameters are, how mixture-of-experts changes the math, and which flagship models still publish size—and which do not.
Jun 29, 2026
For the first time, more U.S. businesses on Ramp pay for Anthropic than OpenAI — 34.4% vs 32.3% in April 2026, with Anthropic quadrupling share in a year while OpenAI grew 0.3%. The viral X thread tied it to loop engineering and Dario Amodei's claim that some Anthropic engineers barely write code anymore. Here is what the index actually counts, what it does not, and why workflow beats chat.
If you have ever read a doc that says "32k context" or "$2.50 per million input tokens" and only half-trusted your mental model, this article is the missing layer: what a token is, why providers count them, and how that connects to limits, bills, and rate limits.
Scope: this is a concepts guide. For dollar math, prompt caching, and agent pipelines, read Caveman skill: token economics and API pricing next.
In daily language we count words. Under the hood, a large language model consumes a sequence of tokens: integer IDs from a fixed vocabulary, produced by a tokenizer (families you will see in papers include BPE, WordPiece, and vendor-specific schemes).
hello might be one token).Why it matters: a "short" line in the editor can still be thousands of tokens once the app attaches system instructions, open files, tool schemas, and prior turns.
Heuristics (English prose, ballpark only): people often use ~4 characters per token, or on the order of one token per ¾ of a word. Do not use heuristics for billing—use the provider's tokenizer or usage dashboard for the model you run.
When you send text to an LLM, the tokenizer breaks it down using Byte Pair Encoding (BPE) or similar algorithms:
For example, the sentence "Understanding tokenization" might become:
["Under", "stand", "ing", " token", "ization"] (5 tokens)[8640, 1302, 287, 11241, 1634] as the model sees itDifferent models use different tokenizers, which is why the same text might be:
Want to see how your text is tokenized? Try our interactive visualizer below to understand token boundaries and compare costs across different models:
| Kind | What counts | Intuition |
|---|---|---|
| Input (prompt) tokens | System prompt, your message, full chat history the client sends, retrieved documents, tool parameters and tool results, images (often a separate budget), etc. | Everything the model must read to respond. |
| Output (completion) tokens | The model's generated text (and sometimes separate billed fields, depending on product). | Everything the model writes. |
Two common surprises:
On frontier models, output is often priced higher per token than input—see each vendor's rate card (e.g. OpenAI, Anthropic).
Output tokens are typically 3-5x more expensive than input tokens:
Reasons for the price difference:
The context window (e.g. 128k or 1M in marketing tables) is the maximum combined budget the model is built to process in a single request: your input plus the room reserved for the reply (how the split is defined depends on the API—read the spec for your model).
| Model | Context Window | Use Case |
|---|---|---|
| GPT-4 Turbo | 128k tokens | ~96,000 words or ~300 pages |
| Claude 3.5 Sonnet | 200k tokens | ~150,000 words or ~470 pages |
| Gemini 1.5 Pro | 2M tokens | ~1.5M words or ~4,700 pages |
| GPT-3.5 Turbo | 16k tokens | ~12,000 words or ~37 pages |
| Llama 3 70B | 8k tokens | ~6,000 words or ~18 pages |
Important: Just because a model can handle 2M tokens doesn't mean you should use them all:
You can still plan in paragraphs and files; the invoice will still speak in tokens.
Different types of content have wildly different token densities:
| Content Type | Characters per Token | Example |
|---|---|---|
| English prose | ~4 chars | "The quick brown fox" = ~4-5 tokens |
| Code (Python) | ~3 chars | def hello(): = ~5-6 tokens |
| JSON data | ~2.5 chars | {"name":"John"} = ~8-10 tokens |
| Chinese text | ~1.5 chars | "你好世界" = ~6-8 tokens |
| Compressed/Base64 | ~1.5 chars | Very token-heavy |
Takeaway: Code and structured data consume tokens faster than you might expect. A 1000-character JSON payload might use 400+ tokens.
Some APIs discount long unchanged prefixes of a prompt when they qualify for cached or reused input (rules differ by provider). The idea: if most of an agent's prompt is a stable system block plus tool definitions, you pay less for that slice on the next call when caching hits. For production patterns, see the Caveman post and your vendor's prompt caching documentation.
Anthropic's Claude offers prompt caching with dramatic savings:
Example savings:
Without caching:
- 50,000 token system prompt + tools = $0.15 per request
- 100 requests = $15.00
With caching:
- First request: $0.15
- Next 99 requests: $0.015 each = $1.485
- Total: $1.635 (89% savings!)
Requirements:
Either way, the scarce resource in aggregate is tokens over time (and provider capacity), which is where rate limits and plan tiers come from.
| Plan Type | Example | Token Budget | Best For |
|---|---|---|---|
| ChatGPT Plus | $20/month | "Unlimited" with caps | Casual users, learning |
| Claude Pro | $20/month | 5x more usage than free | Power users, research |
| API Pay-as-you-go | Variable | Unlimited, billed per token | Production apps |
| Enterprise | Custom | Custom quotas + SLA | Teams, mission-critical |
Hidden truth: Subscription plans have soft limits enforced by:
For developers: If you're building an app, API access gives you:
1. System Prompt Compression
2. Context Management
3. Response Control
4. Caching Everything You Can
Before deploying, test your actual token usage:
OpenAI Models (GPT-3.5, GPT-4):
import tiktoken
encoding = tiktoken.encoding_for_model("gpt-4")
tokens = encoding.encode("Your text here")
print(f"Token count: {len(tokens)}")
Claude Models:
from anthropic import Anthropic
client = Anthropic(api_key="your-key")
count = client.count_tokens("Your text here")
print(f"Token count: {count}")
Online Tools:
Let's calculate costs for a typical customer support chatbot:
Assumptions:
Daily costs:
Input: 1,000 × 5 × 500 = 2.5M tokens
Output: 1,000 × 5 × 200 = 1M tokens
Input cost: 2.5M × $10/1M = $25
Output cost: 1M × $30/1M = $30
Total per day: $55
Monthly: $55 × 30 = $1,650
With optimizations:
Optimized monthly cost: ~$140 (91% savings)
Scenario: Generate docs for 100 repositories
Per repo:
Using Claude 3.5 Sonnet ($3 input, $15 output):
Input: 100 × 55k = 5.5M tokens → $16.50
Output: 100 × 10k = 1M tokens → $15.00
Total: $31.50 for all 100 repos
Alternative with Haiku ($0.25 input, $1.25 output):
Total: $2.63 for all 100 repos
Lesson: Match model power to task complexity. Documentation generation doesn't need Sonnet's reasoning.
Problem: "My prompt is only 50 tokens but I'm billed for 1,500!"
Reality: Your app likely includes:
Fix: Use console.log or debug mode to see full prompts sent to API.
Problem: Each message adds both input AND output from previous turn.
What happens:
Fix: Implement sliding window or summarization after N turns.
Problem: A 10,000-line Python file = ~40,000 tokens
Reality: Most models can't meaningfully process files that large. Attention degrades.
Fix: Use retrieval, chunking, or selective file reading.
Problem: Paying full price when 90% of prompt is identical across calls.
Fix: Structure prompts to put stable content (system prompt, tools) in cacheable prefix.
Understanding how tokenization actually works helps you write more token-efficient prompts.
Most modern LLMs use Byte Pair Encoding, invented for text compression and adapted for NLP:
How BPE builds a vocabulary:
Example of BPE learning:
Initial: ["t", "h", "e", " ", "q", "u", "i", "c", "k"]
Most common pair: "t" + "h" → merge to "th"
Next: "th" + "e" → merge to "the"
Result: Common words become single tokens
Why this matters:
def, import, //) → often 1 token| Model | Vocabulary Size | Implications |
|---|---|---|
| GPT-2 | 50,257 tokens | Smaller vocab = more splits = longer sequences |
| GPT-3/4 | ~100,000 tokens | Balanced for multilingual use |
| Claude | ~100,000 tokens | Optimized for code and reasoning |
| Llama 2 | 32,000 tokens | Smaller = faster, but more tokens per text |
Larger vocabularies:
Smaller vocabularies:
Token efficiency varies dramatically by language:
| Language | Tokens per Word | Example Cost Multiplier |
|---|---|---|
| English | 1.0x baseline | $10 per 1M words |
| Spanish | 1.2x | $12 per 1M words |
| French | 1.3x | $13 per 1M words |
| German | 1.4x | $14 per 1M words (compound words split more) |
| Russian | 1.5x | $15 per 1M words (Cyrillic less common in training) |
| Arabic | 1.7x | $17 per 1M words |
| Chinese | 2.0x | $20 per 1M words (each character often 1+ tokens) |
| Japanese | 2.2x | $22 per 1M words (mixing scripts compounds issue) |
| Korean | 2.5x | $25 per 1M words |
| Thai | 3.0x | $30 per 1M words (no spaces = poor tokenization) |
Why this happens:
Real-world impact:
A Thai company using Claude for customer support might pay 3x more per conversation than a US company with identical usage patterns.
Token efficiency also varies by programming language:
| Language | Chars per Token | Why |
|---|---|---|
| Python | ~3.2 | Concise syntax, common in training |
| JavaScript | ~3.5 | Similar to Python |
| Java | ~2.8 | Verbose syntax, many keywords |
| C++ | ~2.6 | Template syntax, operators |
| JSON | ~2.2 | Braces, quotes, commas each add tokens |
| YAML | ~3.0 | Indentation and colons |
| SQL | ~3.5 | Keywords well-represented |
Optimization tip: When sending structured data to LLMs:
Before compression (expensive):
You are a helpful AI assistant. Please analyze the following
customer feedback and extract key themes, sentiment, and
actionable insights. Be thorough and detailed in your analysis.
Provide specific examples from the feedback to support your
findings. Format your response with clear headings and bullet points.
Customer feedback: [5000 words of feedback]
Tokens: ~1,400
After compression (cheap):
Analyze feedback. Extract: themes, sentiment, actions.
Use examples. Format: headings, bullets.
[5000 words of feedback]
Tokens: ~1,280 (9% savings)
Aggressive compression:
Extract themes+sentiment+actions from feedback below.
Examples+bullets.
[5000 words of feedback]
Tokens: ~1,260 (10% savings)
Key insight: LLMs understand abbreviated instructions. Save verbose explanations for end users.
Instead of sending full chat history, implement intelligent windowing:
def get_relevant_context(messages, max_tokens=4000):
"""Keep most recent + most relevant messages within budget"""
# Always keep system prompt + last 2 messages
core_messages = [messages[0], messages[-2], messages[-1]]
core_tokens = count_tokens(core_messages)
remaining_budget = max_tokens - core_tokens
# Add older messages by relevance score
relevant_old = rank_by_relevance(
messages[1:-2],
query=messages[-1]
)
for msg in relevant_old:
msg_tokens = count_tokens(msg)
if msg_tokens <= remaining_budget:
core_messages.insert(-2, msg)
remaining_budget -= msg_tokens
else:
break
return core_messages
Savings: 40-70% on long conversations while maintaining quality.
Set aggressive max_tokens limits and use streaming to stop early:
response = client.chat.completions.create(
model="gpt-4-turbo",
messages=messages,
max_tokens=500, # Hard cap
stream=True,
stop=["\n\n\n", "---", "In summary"] # Stop sequences
)
for chunk in response:
content = chunk.choices[0].delta.content
if should_stop_early(content):
break # Stop streaming
print(content, end="")
Savings: 20-50% on output costs by preventing rambling.
Route requests to cheapest capable model:
def route_request(query, context):
complexity = assess_complexity(query)
if complexity == "simple":
# Use cheapest model
return call_gpt35(query, context)
elif complexity == "medium":
# Use mid-tier
return call_claude_haiku(query, context)
else:
# Use premium only when needed
return call_gpt4(query, context)
Example routing rules:
Savings: 60-80% on mixed workloads.
Anti-pattern:
# Sends entire knowledge base every time
system_prompt = load_entire_kb() # 50k tokens
Optimized pattern:
# Semantic search for relevant docs only
relevant_docs = vector_search(query, top_k=3) # ~2k tokens
system_prompt = f"Use these docs:\n{relevant_docs}"
Savings: 96% on input tokens
Anti-pattern:
# Send entire codebase
prompt = f"Document this:\n{entire_repo}" # 500k+ tokens
Optimized pattern:
# Process file by file with caching
cached_prefix = """You are a code documentor.
Style: concise, examples, JSDoc format."""
for file in files:
response = generate(
cached_prefix=cached_prefix, # Cached!
prompt=f"Document:\n{file}" # Only new content
)
Savings: 90%+ with prompt caching
Anti-pattern:
# Use GPT-4 for every message
result = gpt4_moderate(message) # $10 per 1M tokens
Optimized pattern:
# Fast filter + selective GPT-4
if simple_filter(message): # Regex, keyword lists
return "safe"
elif likely_violation(message): # ML classifier
return gpt4_moderate(message) # Only edge cases
Savings: 95%+ by filtering obvious cases
Tokens per Request (TPR)
Token Efficiency Ratio (TER)
TER = Useful Output Characters / Total Tokens
Cost per User Interaction (CPUI)
CPUI = Total Token Cost / Number of Interactions
Cache Hit Rate
Hit Rate = Cached Tokens / Total Input Tokens
import anthropic
from datetime import datetime, timedelta
client = anthropic.Anthropic()
def get_token_analytics(days=7):
end = datetime.now()
start = end - timedelta(days=days)
# Fetch usage data
usage = client.usage.list(
start_date=start.isoformat(),
end_date=end.isoformat()
)
total_input = sum(u.input_tokens for u in usage)
total_output = sum(u.output_tokens for u in usage)
total_cached = sum(u.cached_tokens for u in usage)
input_cost = total_input * 3.00 / 1_000_000
output_cost = total_output * 15.00 / 1_000_000
cache_savings = total_cached * 2.70 / 1_000_000
return {
"total_tokens": total_input + total_output,
"input_tokens": total_input,
"output_tokens": total_output,
"cached_tokens": total_cached,
"total_cost": input_cost + output_cost,
"cache_savings": cache_savings,
"cache_hit_rate": total_cached / total_input if total_input > 0 else 0
}
Set up alerts for:
1. Character-level models
2. Multimodal tokenization
3. Adaptive tokenization
4. Token-free billing
Even as tokenization evolves, these principles remain:
Yes. Spaces, tabs, and newlines are tokenized:
Optimization tip: Minimize unnecessary whitespace in prompts:
# Bad: 145 tokens
prompt = """
Please analyze this text:
[Text here]
And provide insights.
"""
# Good: 138 tokens
prompt = "Analyze this text:\n[Text here]\nProvide insights."
Yes, but carefully:
Test before deploying. Quality matters more than token savings.
Common causes:
Solution: Always use official tokenizer for your model:
tiktoken libraryclient.count_tokens() methodBehavior varies by provider:
OpenAI:
maximum context length exceededAnthropic:
Best practice: Track token counts before sending requests.
1 token ≈ 4 characters (English)
1 token ≈ 0.75 words (English)
1 page (500 words) ≈ 650-750 tokens
1,000 tokens ≈ 3-4 paragraphs
Common costs (2026):
GPT-4 Turbo: $10 in / $30 out per 1M
Claude Sonnet: $3 in / $15 out per 1M
GPT-3.5: $0.50 in / $1.50 out per 1M
Claude Haiku: $0.25 in / $1.25 out per 1M
1M tokens ≈ 750k words ≈ 1,500 pages
Tokenizer behavior and plan limits are vendor- and model-specific; always read the current documentation for the product you use.