Patreon gave robots.txt teeth. The creator platform now uses Cloudflare to block known AI training crawlers at the network edge while allowing search crawlers that can send people back to creators. During testing, Patreon says individual training bots fell from thousands of weekly access attempts to zero.
That number exposes the old arrangement. The web spent three decades publishing crawler rules in a public text file and trusting automated clients to honor them. RFC 9309 states the limit plainly: the Robots Exclusion Protocol is a request that crawlers honor rules, and those rules provide no access authorization. AI companies turned that weakness into an extraction strategy. Patreon has replaced etiquette with packet handling.
The trade carries a second consequence. Enforcement requires somebody to identify the machine, infer its purpose, and decide which category gets through. That somebody is Cloudflare, a private network operator sitting in front of more than 20 percent of the web.
the polite web hit its limit
The Robots Exclusion Protocol began as a convention in 1994 and became an IETF standard in 2022. A site publishes groups of allow and disallow rules. A crawler identifies itself with a product token, fetches /robots.txt, matches the applicable group, and follows the paths. The protocol standardizes syntax, redirects, caching, errors, and matching behavior. It cannot lock a door.
RFC 9309 says its rules “are not a form of access authorization.” Its security section directs operators toward HTTP authentication when they need actual control. That design made sense for search indexing. Search engines wanted durable access to pages, publishers wanted referral traffic, and both sides benefited from a lightweight shared grammar.
AI crawling broke the exchange. A training crawler can fetch the work, retain its statistical residue, and return no audience. A retrieval system can answer from the page while keeping the user inside another company’s interface. The publisher still pays for production and delivery. The crawler captures the reusable value.
Patreon had already used robots.txt measures against unauthorized AI training since 2023. Its paywall protected much of the catalog by requiring authenticated access. The risk grew when Patreon expanded free-media discovery through its redesigned Home feed and Quips. More public work improved reach and enlarged the surface available to automated collection.
Patreon’s July 17 announcement describes the fix without euphemism. Cloudflare AI Crawl Control now applies updated policies and enforcement across Patreon. Search crawlers remain allowed. Known training crawlers get blocked before they reach the content. “Consent shouldn’t depend on whether a scraper chooses to behave,” Patreon writes.
three labels now govern the machine audience
Cloudflare’s current taxonomy separates automated traffic into Search, Agent, and Training. Search builds an index intended to help a page appear in results. Agent traffic visits on behalf of a user. Training traffic collects material for model training or fine-tuning. Cloudflare also tracks transaction bots, feed fetchers, social previews, monitoring systems, security scanners, and data collectors.
The categories are useful because “AI bot” has become a garbage label. A podcast reader, shopping agent, vulnerability scanner, model-training scraper, and search indexer create different risks and different value. Blocking every automated request would amputate discovery, feeds, previews, accessibility tools, archives, and useful agents along with the extractive crawlers.
Classification also creates the new failure surface. A crawler can declare one purpose and perform another. Traffic can arrive through a shared cloud platform. Search, summaries, retrieval, and training can share infrastructure or product incentives. Cloudflare combines declared identity with machine learning, behavioral analysis, and network fingerprinting. It can revoke Verified status when a bot abuses the access its identity earned.
That is enforcement with teeth, but the teeth belong to a central intermediary. Cloudflare’s bot directory, detection IDs, behavioral models, and default categories now determine whether a machine reaches a creator’s public work. A mistaken block can erase discovery. A mistaken allow can expose an archive to collection. Small creators cannot audit the classifier from the outside.
Cloudflare’s September 15 policy makes the authority visible. New domains and existing free customers will default to allowing Search while blocking Training and Agent crawlers on ad-supported pages. Mixed-purpose bots such as Googlebot, Applebot, and BingBot will inherit the most restrictive applicable rule. A site that blocks Training can therefore block the mixed crawler entirely unless its owner changes the setting.
Google sits in the ugly middle. Cloudflare says Google accounts for roughly 88 percent of referral traffic and gains access to about twice as much information as leading AI companies because search and AI uses remain coupled. Google offers Google-Extended as an opt-out for some model training and Gemini uses, but Cloudflare argues that a mixed crawler still prevents network-level separation of search from AI consumption. Discoverability becomes leverage over consent.
a creator boundary becomes a CDN constitution
Patreon’s move is still the correct one. A voluntary rule ignored by the relevant actors has ceased to function as consent. Thousands of weekly attempts dropping to zero is direct evidence that network enforcement changes behavior where text-file instruction failed.
The harder task is preventing the cure from becoming another landlord. Cloudflare pitches a progression from blocking to attribution, licensing, Pay Per Use, and agent payments. Its network sees creators, crawlers, referrals, query demand, and access decisions. That position lets Cloudflare create a market. It also lets one infrastructure company become the exchange, customs desk, identity registry, and traffic court for machine access to the web.
The system needs portable policy and auditable identity. Site owners should be able to export crawler classifications and enforcement logs. Bot operators need signed, purpose-specific identities that survive proxies without collapsing every downstream user into the platform that carried the request. Decisions should expose evidence, expiry, and appeal paths. Creator policy must remain usable outside one CDN dashboard.
Cloudflare is experimenting with pieces of that model. BotBase records known automated clients and their behaviors. Verified bots must represent themselves honestly and avoid abusing access. Proposed Content-Signal fields can state search=yes, ai-train=no, and a permitted use level such as reference. A Forwarded header can preserve the operator behind a proxy and carry a declared use. These signals remain preferences until an enforcement layer acts on them.
the open web is becoming a controlled port
Patreon’s decision lands during a broader reversal. The early web treated automated reading as a source of circulation. Search crawlers mapped pages and sent readers back. RSS moved full text while preserving subscription relationships. Archives copied pages for public memory. The crawler was part courier, part librarian.
Training and answer engines changed the economics. Cloudflare reports that automated agents and bots now generate more than half of web requests. It says over half of AI crawler traffic re-fetches unchanged pages, while some heavily crawled industries have lost up to 40 percent of human traffic in under a year. Those figures come from a company selling the enforcement and measurement layer, so they deserve scrutiny. The direction is difficult to dispute: machines consume a growing share of published work, and fewer humans arrive at the source.
A controlled port can protect the people who built the goods. It can also decide which ships count as trade, piracy, research, or public service. Patreon has made one defensible call: search may enter, training may not. The next fight concerns who writes those labels, how they are verified, and whether creators can carry the rules somewhere else.
robots.txt survived because it was tiny, public, legible, and host-controlled. Its weakness was voluntary compliance. The replacement should preserve those virtues while adding authentication and enforcement. Otherwise the web will solve AI scraping by trading a polite public standard for a handful of opaque private border guards.