explainx / blog
Cloudflare and Vercel now serve clean Markdown to AI agents via Accept: text/markdown headers, cutting payload size up to 99%. Here's how content negotiation for agents works, real adoption data, and the cloaking debate.
Jul 22, 2026
Jack Dorsey announced Buzz on July 21, 2026 — a self-hostable, open-source workspace where humans and AI agents share one identity system across chat, Git, and workflows. Every message and code event is a signed Nostr event. Here's what's real, what's early, and why it matters for anyone running Claude Code, Codex, or Goose on a team.
Jul 21, 2026
A DeepLearning.AI course on turning raw data into knowledge graphs with multi-agent systems collided on X with Linear's launch of Loops — recurring autonomous agent workflows for bug triage and doc updates. The resulting debate over graphs versus loops for agent orchestration echoes a decades-old software engineering argument that keeps resurfacing under new names.
Jul 16, 2026
DoorDash opened a limited beta for dd-cli, a command-line interface that lets AI agents search restaurants, find deals, and complete real payments from the terminal. explainx.ai maps the apps-for-agents thesis, macOS-only waitlist constraints, Paul Graham memes vs skepticism, and checkout safety next to today's Codex $HOME news.
A growing number of websites now answer a request differently depending on one HTTP header. When a client sends Accept: text/markdown instead of the usual Accept: text/html, some servers respond with a stripped-down Markdown document instead of a full web page — no navigation, no JavaScript, no CSS, just the content. Cloudflare shipped this as a zone-level feature called Markdown for Agents, and Vercel documented the same pattern natively for Next.js sites. The pitch is straightforward: AI agents operating under token budgets get the same information for a fraction of the payload.
The mechanism is old — HTTP content negotiation via the Accept header has existed since the earliest web standards, used for decades to negotiate image formats and localized content. What's new is applying it specifically to AI agents, and that's where the story gets more contested than the token-savings pitch suggests. This is part of the same broader shift toward generative engine optimization — but unlike most GEO techniques, this one has drawn direct, public pushback from Google and Microsoft, and a genuine security concern from the SEO community.
| Question | Answer |
|---|---|
| What is it? | HTTP content negotiation — Accept: text/markdown gets a Markdown response instead of HTML |
| Who ships it? | Cloudflare (zone-level toggle) and Vercel (Next.js-native pattern) |
| Reported savings? | Up to 99% payload reduction (Vercel: ~500KB HTML → ~3KB Markdown) |
| Do agents actually request it? | Only 3 of 7 tested coding agents do, per a Feb 2026 Checkly survey |
| Which agents send the header? | Claude Code, Cursor, OpenCode |
| Which don't (yet)? | Codex, Gemini CLI, GitHub Copilot, Windsurf |
| Is it risky? | Yes — SEO practitioners flag a real cloaking/prompt-injection concern |
| Do Google/Microsoft endorse it? | No — both have publicly questioned its value |
HTTP has always supported multiple representations of the same resource. A browser's Accept header typically lists text/html first, signaling it wants a rendered web page; an image request might specify Accept: image/webp, image/avif, image/*. Servers read that header and choose which representation to return — this is content negotiation, and it's been part of the HTTP specification since RFC 2616.
Markdown for Agents extends the same mechanism to a new content type. When a client's Accept header includes text/markdown, a properly configured server or edge layer intercepts the request before it hits the normal HTML rendering path and returns a Markdown document instead — typically with navigation, sidebar widgets, embedded scripts, and decorative markup stripped out, leaving the actual prose, headings, links, and code blocks intact. The response carries Content-Type: text/markdown so the client knows what it received.
Cloudflare implements this at the edge, as a per-zone setting that converts HTML to Markdown on the fly for any origin sitting behind its proxy — the origin server doesn't need to change anything. Vercel's approach is framework-native: a rewrite rule in next.config.ts matches on the Accept header and routes matching requests to a dedicated handler that returns the Markdown representation directly, giving the site owner full control over exactly what that Markdown contains rather than relying on an automated HTML-to-Markdown conversion pass.
The efficiency argument is the strongest part of the pitch, and it's grounded in something structurally true about HTML: markup itself consumes tokens without conveying information an LLM needs to answer a question. A <div class="sidebar-nav flex items-center"> wrapper, repeated across every page of a site, adds real token cost on every single request an agent makes — cost that scales with how many pages an agent has to read to complete a task.
Vercel's own reported figure — a blog page dropping from roughly 500KB of HTML to about 3KB of Markdown, a 99.37% reduction — is a genuinely striking number, but it's also close to a best case: a content-heavy page with a lot of surrounding template markup relative to actual prose. More typical estimates circulating in developer discussion of the pattern cluster closer to an 80% reduction for average content pages, which is still substantial. The practical effect, as several implementers have put it, is the difference between an agent that can read five pages of documentation within its context budget and one that can read twenty-five.
The efficiency case only pays off if the requesting client actually sends the header — and current data suggests most don't, yet. A February 2026 survey by Checkly tested seven widely used coding agents against sites implementing Markdown content negotiation and found real, near-even split in adoption:
Sends Accept: text/markdown? | Agents |
|---|---|
| Yes | Claude Code, Cursor, OpenCode |
| No (as tested) | OpenAI Codex, Gemini CLI, GitHub Copilot, Windsurf |
That's a meaningful gap — three out of seven major coding-agent tools currently negotiate for Markdown, while four default to requesting plain HTML the same way a browser would. Some of the agents that do send the header use HTTP quality factors (q= values) to express a soft preference rather than a hard requirement, while others rely purely on header ordering, meaning implementation details even among the adopting agents aren't fully standardized yet. For a site owner deciding whether to build this, the honest framing is: the technique benefits a specific, currently narrow slice of AI-agent traffic today, with the expectation — not the guarantee — that adoption broadens as more agent frameworks catch up.
The most substantive criticism of Markdown for Agents isn't about efficiency at all — it's about trust. SEO consultant David McSweeney raised the concern directly: because the Accept: text/markdown header is visible to the origin server before any content is returned, it functions as a reliable signal that "this request is probably from an AI agent." A server that wanted to behave differently for agents than for humans now has an easy, standards-compliant way to do it — return one thing to a browser, something else entirely to anything that asks for Markdown.
That's structurally identical to classic search-engine cloaking, the long-banned practice of showing crawlers different content than human visitors see, just retargeted at AI agents instead of Googlebot. Analysis from queryburst.com sharpened the concern into a specific attack scenario: a malicious or compromised site could embed instructions in the Markdown response — content specifically crafted for an AI agent's parser — that a human reviewer visiting the same URL in a browser would never see and therefore never catch. Because AI agents are increasingly trusted with real actions (browsing on a user's behalf, executing code, managing credentials), a parser that implicitly trusts whatever text a page returns is a meaningful target for indirect prompt injection delivered through exactly this channel.
It's worth being precise about scope here: this isn't a vulnerability specific to Cloudflare's implementation or Vercel's pattern — it's a property of any system that changes its response based on a machine-readable signal indicating "an AI agent is asking." Cloudflare and Vercel didn't invent the risk; they made the underlying technique — content negotiation — dramatically easier to deploy at scale, including for site operators who wouldn't otherwise have built custom bot-detection logic to serve different content to different requesters.
Neither Google nor Microsoft has issued formal policy guidance on whether Markdown-for-agents content negotiation counts as cloaking under existing spam policies. But representatives from both companies have voiced skepticism about the practice on its merits.
John Mueller, Google's long-running search advocate, has argued that the premise undercuts itself: large language models have already been trained on and can parse ordinary HTML pages perfectly well, since HTML has been part of the open web — and LLM training corpora — from the start. In that framing, a dedicated Markdown representation solves a problem that mostly doesn't exist for models that already handle markup natively; the real audience benefiting is agentic tools operating under strict runtime token budgets, a narrower use case than "AI in general."
Fabrice Canel of Microsoft/Bing raised a more operational objection: Bing's crawler will fetch both the HTML and Markdown representations of a page anyway, specifically to verify the two are equivalent and neither is being used to serve manipulated content selectively. From Microsoft's side, that turns the pattern into extra crawl load rather than the efficiency win it's pitched as — the savings accrue to the AI agent making the request, not to the search engine verifying content integrity on the back end.
Anyone considering implementing Markdown content negotiation should treat it as more than a formatting exercise. A few checks worth making before shipping it:
.md suffix, for instance) needs clear indexing guidance so it isn't crawled and indexed as a separate, competing page against the canonical HTML version.Accept header, or it risks serving a Markdown response to a browser or an HTML response to an agent that explicitly asked for something else.Adoption figures, quotes, and industry reaction reflect publicly available reporting as of July 22, 2026, including Checkly's February 2026 agent survey and public commentary from Google's John Mueller and Microsoft's Fabrice Canel. Agent behavior and header support change quickly — verify current adoption before making architecture decisions based on the figures above.