Last week’s Patreon crawler piece covered the first move: creator consent left robots.txt and moved into Cloudflare’s edge enforcement. The new development is Cloudflare’s July 1 bot-policy rollout becoming the general access model for all customers, with separate controls for Search, Agent, and Training traffic, September 15 defaults for ad-supported pages, and a narrower meaning for Verified bots.
Cloudflare is making AI crawlers choose a lane. That sounds like dashboard housekeeping until you stare at the mechanism. The web’s machine audience is being sorted by declared purpose, monetization context, crawler identity, and downstream content use. Search can still enter by default. Training and Agent traffic hit a harder boundary on pages with ads. Mixed-purpose crawlers inherit the strictest applicable rule.
purpose is now an access primitive
Cloudflare’s old consumer-grade control was blunt: block AI bots. The new model splits automated traffic into three headline AI behaviors. Search means crawling that builds indexes or retrieval databases. Agent means user-directed automation visiting a page on behalf of a person. Training means collecting material to train or fine-tune models. Cloudflare’s fuller taxonomy also tracks transaction bots, data collection, security testing, SEO, ads verification, social previews, feed fetching, and monitoring.
That taxonomy matters because “AI bot” has become a junk drawer. A podcast client, search crawler, personal shopping agent, model-training scraper, uptime monitor, and link-preview fetcher all touch public pages with different social contracts. The old binary let vendors hide behind category fog. The new controls make purpose the unit of admission.
Cloudflare says all customers, including Free-tier customers, can now tune AI traffic by Search, Agent, and Training. The operational bite arrives on September 15, 2026. New domains onboarding to Cloudflare will block Training and Agent bots on pages that display ads while allowing Search. Mixed-purpose crawlers that combine Search and Training will be handled by all of their behaviors, using the most restrictive applicable rules.
the ad slot became a policy sensor
The ad-page default is the nasty little systems detail. Cloudflare treats an ad as evidence that the site owner expected human attention to arrive there. That lets the network enforce different machine rules on monetized pages than on the rest of the site. The page’s business model becomes part of the access decision.
That is defensible. If an answer engine or training crawler consumes an ad-supported article without sending a reader, it drains the economic circuit that paid for the article. Search still gets default access because it plausibly returns visitors. Agent and Training traffic get fenced because their value path often bypasses the publisher.
It also creates a fresh control surface. Ad detection, crawler classification, and default rule inheritance now decide whether automated traffic touches a page. A publisher can opt out, but the default writes the policy for whoever does nothing. Defaults are governance with a friendly settings label.
Verified no longer means welcome
The sharpest change is the Verified-bot label. Cloudflare’s updated docs define a Verified bot as one that identifies itself honestly and avoids abusing the access that honesty earns. Verification can come through cryptographic Web Bot Auth, a published IP list plus stable user-agent, or reverse DNS. Behavior still matters: robots directives, request rates, and owner preferences remain part of the bargain.
Previously, Verified often functioned like default admission. Cloudflare’s July language changes the meaning. Verified now makes a bot allowable inside its relevant category. If a site allows Search, a Verified Search crawler can enter. If the same crawler also performs Training and the site blocks Training, the stricter category can block it. Verification becomes passport control, then customs inspection.
That solves one abuse pattern and introduces another political problem. The ability to revoke Verified status across Cloudflare’s network gives crawler compliance real consequences. It also concentrates a huge amount of judgment inside BotBase, Cloudflare’s directory of tracked bots and agents. A private infrastructure company is defining the recognized machine identities of the web.
content use is being smuggled into the protocol layer
Cloudflare is also testing a use field for Content Signals in robots.txt: immediate, reference, or full. Immediate means interact without storing and reusing. Reference means index, excerpt, and link back. Full means summarize and reproduce. Cloudflare says managed robots.txt users who already publish search=yes,ai-train=no will receive use=reference.
That is a weird little grammar for the next web. It gives publishers a way to say that indexing and linking are permitted while full reproduction is not. Like the rest of robots.txt, the signal is a preference until an enforcement layer acts. Cloudflare closes that loop by tracking content use in BotBase and threatening Verified-status loss for bots that abuse the signal. Today, bots that reproduce in full cannot have Verified status.
There is a cleaner future hiding inside this messy rollout: portable content-use policy, signed bot identity, transparent crawler categories, and logs a site owner can inspect outside one CDN. The current version is useful because it works at the edge. It is dangerous for the same reason.
transitive trust is the agent problem arriving early
Cloudflare’s transitive-trust proposal points at the agentic web problem before the agentic web has standards sturdy enough to deserve the name. A request may come from a platform that hosts automation for thousands of downstream users. The site may trust the platform, distrust some operator riding through it, or want a use-level promise preserved across proxies.
Cloudflare proposes using the existing Forwarded header to carry operator information and a content-use declaration such as Forwarded: for="openai";use="reference". That is elegant plumbing if everyone behaves. It is also exactly where the politics get ugly. The intermediary becomes the party expected to preserve identity, classify purpose, and attach consequences when a downstream operator lies.
This is where machine access stops looking like crawling and starts looking like customs brokerage. Every request wants papers. The network wants a known operator. The publisher wants a use limit. The crawler wants enough ambiguity to keep access broad. The CDN becomes the checkpoint because the origin server cannot maintain the classifier alone.
the replacement for the polite web needs escape hatches
Cloudflare’s move is directionally correct. Voluntary crawler etiquette collapsed once model training and answer interfaces broke the referral exchange. Publishers need enforceable distinctions between search, user-directed automation, training, summaries, feed fetching, ads verification, and outright scraping. Pretending all automated access has the same bargain is how extractive platforms get a free buffet while everyone else pays bandwidth and labor.
The replacement cannot be a private black box with a nicer UI. Publishers need exportable policy, auditable decisions, appeal paths, portable bot identity, and logs that survive a CDN migration. Bot operators need purpose-specific crawlers instead of one mixed-use identity dragging search, RAG, training, and product features through the same pipe. Regulators should understand that crawler defaults are market structure now. Search access, agent access, and training access decide who receives traffic, who receives data, and who gets to build on the public web without permission.
The old robots.txt bargain was weak because it lacked enforcement. Its virtues were legibility, portability, and origin control. Cloudflare is supplying enforcement. The fight now is making sure the new machine border keeps the old web’s legibility before the dashboard becomes the constitution.