One year after Content Independence Day, Cloudflare stopped pretending every “AI bot” is the same animal. The July 1, 2026 post — Your site, your rules: new AI traffic options for all customers — ships a pragmatic taxonomy: Search, Agent, and Training, with Free-tier controls and a hard calendar date: September 15, 2026.
Hacker News latched onto the load-bearing footnote Simon Willison highlighted: multi-purpose crawlers that combine Search with Training — Googlebot, Applebot, BingBot — will be blocked when you block Training, because the most restrictive applicable rule wins. That is not a side effect. It is the point of Cloudflare’s transparency push — and it can quietly delist you from classic search if you leave legacy “Block AI bots” on and ignore the mail.
This explainx.ai guide is the operator checklist: what the categories mean, what changes on Sept 15, how Verified/BotBase/content-use fit, and how this sits next to llms.txt, agent content negotiation, and Cloudflare’s monetization gateway experiments.
TL;DR — What People Are Asking
| Question | Answer |
|---|---|
| New controls? | Search / Agent / Training (all plans) |
| Deadline? | September 15, 2026 new defaults |
| Ad pages default? | Block Training + Agent; allow Search |
| Googlebot risk? | Block Training → can block Googlebot |
| Who’s affected? | New domains + zones without explicit prefs (verify yours) |
| Opt out? | Yes — Security settings before Sept 15 |
| Verified bots? | No longer “always allow” |
| Content use? | immediate / reference / full |
| Pay-per-crawl? | Still gated for many — don’t bank on it |
The Taxonomy (Stop Saying “AI Bot”)
Cloudflare’s argument: AI is now in search answer engines, chat fetchers, and training crawlers — so the useful question is behavior, not branding.
| Class | What it does | Expectation for the site |
|---|---|---|
| Search | Indexes to answer later | Referrals / equitable compensation |
| Agent | Real-time job for a human | Often a person waiting; chat/browser agents |
| Training | Absorbs content into model weights | Permanent capability gain for the lab |
BotBase (Enterprise) tracks more behaviors — Transact, Data Collection, SEO, Ads Verification, Feed Fetching, Monitoring, Security Testing — but the three AI-centered knobs are what every customer can manage today.
Cloudflare’s ask of bot operators: split crawlers by purpose. A company that searches, agents, and trains should not hide behind one UA string. Until they do, site owners get a blunt instrument that treats multi-purpose bots by all declared behaviors.
September 15, 2026 — Defaults With Teeth
For new domains onboarding to Cloudflare:
- Training — blocked by default on pages that display ads
- Agent — blocked by default on ad pages
- Search — allowed by default
Ads are treated as a signal that a human should land and see monetizable attention. Training and Agent traffic that substitutes for that attention get the cold shoulder; Search is framed as the behavior that still funnels visitors.
The multi-purpose trap
Also on Sept 15: crawlers that combine Search + Training are allowed/blocked according to all behaviors. Most restrictive rule wins. Cloudflare names Googlebot, Applebot, and BingBot explicitly.
So if you:
- use legacy Block AI bots, or
- select block Training in the new UI,
you may also block the bots that keep you in Google/Bing/Apple search. That is catastrophic for SEO / GEO sites that only meant to stop GPTBot-style training scrapes.
Action: open Security settings, record an explicit preference for Search / Agent / Training, and decide whether you opt out of the new default configuration for multi-purpose Search+Training crawlers. Doing nothing is still a decision.
Verified Bots Are Not a Free Pass Anymore
Previously, Verified ≈ default allowed. Now Verified means the bot is honestly labeled and allowable within its category. Non-verified remain default-blocked in Bot Fight Mode style flows, but Verified Search traffic only flows if you allow Search.
Abuse the Content Signals or reproduce at use=full without permission and you can lose Verified status — Cloudflare’s deterrent story across the ~20%+ of the web behind their network.
Content Use: immediate / reference / full
Bot Management customers can combine classification with content use:
| Level | Meaning |
|---|---|
| immediate | Interact; store/reuse nothing |
| reference (default) | Index, excerpt, link back |
| full | Summarize and reproduce |
robots.txt gains an optional Content-Signal field. Managed robots.txt that already said search OK / train no can now add use=reference:
User-agent: *
Content-Signal: search=yes,ai-train=no,use=reference
Allow: /
Signals are preferences, not enforcement — same honor-system critique HN made about robots.txt generally. Pair with real allow/block rules if you care.
Transitive Trust and Forwarded
Agents increasingly arrive through platforms, not the lab’s own IP. Cloudflare proposes extending Forwarded (RFC 7239) so site policies can follow the end operator:
Forwarded: for="openai";use="reference"
Trust becomes portable — and revocable — across intermediaries. Whether small privacy-preserving agents can live in that world is an open question Cloudflare itself flags (private rate limiting as a future building block).
What HN Is Actually Debating
- Google’s search+train coupling — predatory or just legacy architecture? Blocking Training forcing Googlebot out is the only leverage many site owners feel they have.
- Cloudflare as MITM / CDN politics — separate from the taxonomy; don’t let that thread distract from your zone settings.
- Anubis / PoW alternatives — useful if you reject CF’s challenge UX; different product.
- Pay-per-crawl theater — “request access” with silence; the taxonomy shipped, the cash pipe mostly didn’t.
- Both sides of the arms race — CF sells AI products and bot walls; incentive critique is fair and does not change your Sept 15 checklist.
Training scrapes also feed the dataset economy we covered around Hugging Face The Stack — site-level controls are one layer; corpus licensing is another.
Operator Checklist Before September 15
- Inventory — Does this zone show ads? New domain or old? Any legacy Block AI bots?
- Decide per class — Search allow? Agent allow on product pages even if ads? Training block everywhere?
- Google/Bing/Apple — If you need classic search, do not leave Training-block rules that also catch multi-purpose bots unless you explicitly accept that trade.
- Agent traffic — If ChatGPT-User / Claude fetch / browser agents drive demos or support, allow Agent on the URLs that matter (or you will break live answers on ad pages).
- robots.txt — Align Content-Signal with reality; don’t claim
use=referencewhile your marketing site expects full reproduction. - Document — Write the policy in one internal doc so the next SEO hire doesn’t flip a toggle and delist you.
- Monetization path — If you want paid access, track x402 / micropayment gateway experiments separately from free-tier bot toggles.
Honest Limitations
- Classification depends on Cloudflare’s bot directory — mislabels happen.
- Honor-system robots.txt does not stop bad actors.
- Defaults target ad pages — understanding “what counts as an ad” on your stack matters.
- Enterprise BotBase visibility is not Free-tier feature parity.
- Pay-per-crawl is not a reliable revenue plan today.
- Blocking Training will not stop labs that already have your content or buy scrapes elsewhere.
Worked Example: Three Zone Policies
Publisher with ads + SEO dependence
Allow Search. Block Training on all pages. Allow Agent on article pages if AI answers drive measurable referral clicks; block Agent on pure ad-landing URLs if scrapers cannibalize attention. Explicitly opt out of any default that treats Googlebot as Training-blocked unless you have verified Googlebot is not classified multi-purpose for your zone.
SaaS docs / developer portal
Allow Search and Agent — agents reading docs are customers. Block Training if you sell proprietary guides; allow Training if you want models to “know” your public API (many API companies do).
Membership / paywalled essays
Block Training. Be careful with Agent — live fetches can bypass paywalls for end users. Prefer use=immediate preferences plus hard rules on gated paths. Pair with markdown-for-agents negotiation if you offer a paid clean-text feed.
How This Interacts With GEO
If your growth loop is “get cited in AI answers,” Search and often Agent are the traffic that creates those citations. Blind Training blocks that also catch multi-purpose search crawlers are an SEO own goal. Map your GEO strategy to the three toggles before Sept 15 — not after Google Search Console traffic falls off a cliff.
Related on explainx.ai
- Cloudflare monetization gateway — x402 / MCP micropayments
- llms.txt specification guide
- Markdown for agents — content negotiation
- What is SEO / GEO?
- Hugging Face The Stack v3 — training data scale
- Closed-source AI vs local open-source alternatives
- Open weights American AI leadership letter
Primary sources: Cloudflare Blog — Your site, your rules · Cloudflare Changelog — AI traffic options · Hacker News discussion of the announcement.
Defaults and product names are accurate as of July 26, 2026. Re-read Cloudflare’s Security docs before September 15 — enforcement details can shift.