Multimodal Vision Token Budget: How Images Consume Context Windows

Multimodal context is one wallet. Text instructions, tool JSON, chat history and every image tile draw from the same limit. Teams that budget text-only prompts get surprised when three screenshots crowd out the system prompt.

Quick answers

How vision encoders bill context

Most frontier multimodal models do not feed raw pixels directly into the text transformer. A vision tower converts image patches into embeddings aligned to the language model's token space. APIs expose that cost as vision tokens or image tokens so developers can reason about limits.

High-resolution pages with small fonts push encoders to more tiles. That is why a single desktop screenshot can exceed an entire policy document in token cost. Google's EmbeddingGemma 2 paper and Gemma vision docs describe configurable vision token budgets (70–1120 soft tokens per image); generative APIs follow the same tradeoff under different names.

A ribbon passing through a window with its far end fading, showing how text and vision inputs share one context window

For narrative background on shared limits, our multimodal AI guide warned that image tiles can consume thousands of tokens each while still sharing one window with text.

Drag the lab sliders: notice how six tiles per image across three frames can dominate a 128K window before the assistant speaks.

Provider patterns (planning, not quotes)

Always read the current doc for the model you deploy. OpenAI's vision guide documents patch- and tile-based billing (for example, 170 tokens per 512px tile on gpt-4o at detail: high, plus a per-image base). Patterns as of late 2026:

• Tile or patch budgets tied to max edge length.
• Detail or quality flags that step tile count up or down.
• Video billed per frame or sampled frames — agent loops that screen-record burn budget fast.
• PDF page renders treated like images when you rasterize.

When OpenVLHarness agents zoom crops, each zoom is another vision charge. Design harnesses to keep references addressable without resending full frames every turn.

Agent and computer-use implications

Computer use and Codex computer use workflows tempt agents to attach full-screen PNGs each step. Costs explode:

An outlined hand lowering a cream tile into a compartmented tray, suggesting cropped UI captures instead of full-screen vision tokens

1. Prefer structured state — accessibility tree, DOM, snapshot refs.
2. Crop aggressively to the widget under test.
3. Diff frames — send stills only when the UI changed materially.
4. Cap history — retire old screenshots from context; store them externally and reference IDs (thin prompts, thick artifacts).

OpenVLHarness multimodal memory exists partly so agents do not re-embed identical crops.

Budgeting worksheet

Use this order when estimating a production prompt:

1. System + tools — static prefix (tool schema design).
2. Retrieved text — RAG chunks with reranker cap (reranker guide).
3. Conversation text — compact or summarize (Headroom-style compression if needed).
4. Vision inputs — tiles × frames × tokens per tile.
5. Reserve for assistant output — never spend 100% on input.

If step 4 leaves less than your minimum output reserve, drop frames before dropping safety instructions.

Local and edge stacks

On-device MLX and Gemma vision updates still respect finite RAM. Vision tensors add activation memory beyond weight footprint. When you run EmbeddingGemma 2 locally, lowering vision token budget is a first-class knob.

Evaluation and monitoring

Track per session:

• Vision tokens per user turn (p50 / p95)
• OCR failure rate when you downscale to save tokens
• Tool-call success before vs after screenshot diet

Regression here looks like "the model forgot the policy" when in fact the policy was truncated.

Tile math without vendor secrets

Exact formulas differ, but planning math looks like:

vision_tokens ≈ n_images × tiles_per_image × tokens_per_tile

Where tiles_per_image grows with max(width, height) after the API's resize policy. tokens_per_tile is model-specific — read the card.

Example for capacity planning only (tile counts are illustrative; see your provider's calculator): three UI screenshots at high detail with six 512px tiles each and 170 tokens per tile on a model like gpt-4o → 3 × (85 + 6 × 170) ≈ 3,285 vision tokens before any text, using OpenAI's published base-and-tile table. Add a 6k-token tool schema and 20k tokens of chat history and a "128k" model is already tight.

Always leave output headroom. If you target 2k tokens of assistant JSON, your input budget is not 128k — it is 128k minus 2k minus safety margin.

Detail modes and cropping playbook

Before you spend vision tokens, ask whether RAG text already contains the fact. Vision is for layout, color, charts and unreadable PDF scans.

Video and multi-frame agents

Screen recordings for computer use multiply costs linearly with sampled frames. Prefer:

• Keyframe sampling on UI change detection
• Event stream (click, type, navigation) with a single confirm screenshot
• Lightpanda-style headless browsers that expose DOM cheaply

For robotics and physical AI, camera streams differ — see OpenVLHarness for artifact memory instead of resending full video tensors into chat context.

Batching and serving angle

Multimodal prefill is heavier than text-only. When you self-host with vLLM vs SGLang, vision inputs affect scheduling and KV cache residency. Batch requests with large images together cautiously — one huge screenshot can delay unrelated text chats unless you use separate queues.

Pricing and product implications

Context window pricing often lists input tiers. Vision tokens usually bill as input tokens at the same per-million rate unless the vendor lists a vision surcharge. Product managers should model p95 screenshot counts per workflow, not demo happy paths.

Customer-facing copilots that let users attach unlimited images need server-side caps and user-visible counters — otherwise support tickets look like model stupidity when context truncated the policy.

Testing checklist

1. Fixture set of 10 real UI captures at native resolution.
2. Measure vision tokens reported by API per fixture at each detail level.
3. Run golden questions with text-only, low detail, high detail; compare accuracy delta.
4. Automate regression in CI when you change resize code in the client.

Research and embedder angle

EmbeddingGemma 2 exposes truncatable embedding dimensions and adjustable vision token budgets for retrieval, not generation — but the same intuition applies: multimodal stacks expose knobs because one-size-fits-all waste is enormous.

For science PDFs, consider layout parsers (PixelRAG) that store visual tiles in a vector DB instead of stuffing every page into the chat window each turn.

Fine-tuning and eval notes

Vision-heavy fine-tunes (SAM 3 workflows) use different data pipelines than chat screenshots. Do not extrapolate tile counts from segmentation crops to Copilot-style chat uploads.

When you run RULER-style long-context evals, multimodal fixtures are still rare — supplement with your own screenshot suite.

Hardware footprint

Vision prefill increases activation memory. On laptop MLX runs, watch wired memory when you attach multiple images — the OS may thrash before the API returns a polite context error.

Common mistakes

• Treating marketing context window numbers as usable space (decoded pricing).
• Sending duplicate screenshots across subagents in parallel (worktree parallelism).
• Using vision where text logs suffice (API JSON errors, stack traces).

Related reading

• What is multimodal AI?
• KV cache and memory
• Context window pricing decoded
• Vercel agent-browser accessibility snapshots
• OpenVLHarness visual agent memory
• Thin prompts and thick artifacts

Mapping APIs you may already use

Several 2026 agent stacks expose vision indirectly:

• Chat Completions with image_url parts — count tiles per attachment.
• Responses API — multimodal inputs may report usage breakdowns; log them per trace.
• Gemini files API — uploads may persist; still bill when referenced in generation.
• Claude PDF uploads — pages rasterize; prefer text extraction when the PDF is digital.

When you compare OpenVLHarness presets across Qwen-VL and GPT-family models, token reports differ even on the same PNG. Normalize metrics in your gateway rather than trusting one vendor's demo screenshot.

For coding agents, combine this guide with DESIGN.md and UI capture practices so designers know which screens will be screenshotted during agent reviews.

If you operate a gateway, expose vision_tokens and text_tokens separately in usage webhooks. Finance and reliability teams ask different questions from the same trace, and merged totals hide screenshot regressions after a client app update.

On-call playbooks should list "context truncation" as a first hypothesis when multimodal features launch. The fix is often operational — fewer frames, smaller crops — not a model swap. Record before-and-after token counts in the incident doc so the next team does not repeat the same attachment pattern. That single habit prevents expensive model upgrades that only mask bad attachment hygiene.

Builders shipping agent sandboxes should meter screenshot volume per session the same way they meter shell minutes.

Enterprise rollout

Document per-role limits: support tier 2 may attach five images; automated agents may attach none without human approval. Pair limits with redaction policies when screenshots contain PII.

Train internal users that "the model ignored the policy" often means the policy text was truncated — not refusal. Show token usage in internal UIs when you roll out vision features.

Article accurate as of November 9, 2026. Verify vision token tables on each provider's model card before capacity planning.