Cloudflare lets site owners choose which AI bots can access their content

One year after launching its "Content Independence Day" initiative, Cloudflare is rethinking how website owners should govern AI traffic to their sites. The company's original "Block AI Bots" feature offered a single-purpose solution: stop crawlers trained on model data. But conversations with customers and observation of the market revealed that approach was inadequate. Website owners wanted the ability to distinguish between different kinds of AI automation—some beneficial, some extractive—rather than blocking everything. Cloudflare's new framework defines three use cases. Search encompasses any crawling that indexes or collects content to answer questions later; the assumption is that site owners should receive referral traffic or compensation. Agent covers real-time automated behavior acting on a person's behalf, such as ChatGPT User-Agent fetches or browser-use agents like Gemini or Claude driving Chrome; these bots visit a web application to complete a task immediately. Training refers to crawlers that absorb content into a model's underlying architecture. Each of these three categories can now be managed independently on Cloudflare, available to all customers including the Free tier. Beyond these three, Cloudflare's taxonomy also tracks other bot behaviors—ads verification, feed fetching, SEO crawling, security testing, data collection, and monitoring—a full accounting that reflects the evolved state of automated web traffic. The company is now calling on bot operators to separate their crawlers by purpose, arguing that transparency benefits site owners by clarifying why a given crawler is visiting and enabling finer-grained access control. Starting September 15, 2026, Cloudflare will apply new defaults to all new domains onboarding to the platform. On pages that display ads, Training and Agent bots will be blocked by default, while Search will remain allowed. The reasoning is straightforward: an ad signals that a website owner intended human attention to land there and monetize it; bots that prevent that attention undermine the site owner's business model, while Search traffic typically funnels visitors back to the site. Multi-purpose crawlers—those that combine Search with Training, including Google, Apple, and Bing's bots—will be subject to the most restrictive rule a customer has applied; if a site owner has selected to block Training, those multi-purpose crawlers will be blocked entirely. Cloudflare will notify customers of the upcoming change and allow them to opt out by changing their settings anytime before September 15. For Enterprise customers, Cloudflare is also launching BotBase, a new dashboard feature that provides a searchable database of all known bots, their classifications across the taxonomy, and their content-use behaviors. This visibility tool will later expand to direct control, allowing Enterprise customers to manage known bots in real time. Cloudflare is also introducing a new optional field in robots.txt, part of its Content Signals standard, called "use" which allows site owners to express one of three content-reuse preferences: immediate (interact but store nothing), reference (index, excerpt, and link back), or full (summarize and reproduce). All customers already using Cloudflare's managed robots.txt will now have use=reference added automatically alongside the existing search=yes,ai-train=no signals. Bots that abuse these signals will lose their Verified status and cease being allowed on the platform, creating an incentive for operators to honor site owner preferences and maintain compliance with the emerging standard.

Cloudflare's update reflects a fundamental shift in how the web community views AI bot traffic. One year ago, the company launched a stark binary choice—"Block AI Bots"—in response to what it called an existential threat: AI systems scraping content for training without compensating creators. But a year of customer feedback revealed the flaw: that approach harmed small websites most, forcing them to choose between discoverability via search engines and protection from AI training, a Faustian bargain that entrenched large incumbents who control both search and AI infrastructure. The new three-category taxonomy (Search, Agent, Training) and the "content use" framework acknowledge that not all AI automation is predatory—real-time agent behavior and search indexing can drive value back to site owners—while still giving owners the tools to prevent their content from being absorbed wholesale into proprietary models without compensation. By setting Search as allowed-by-default and Training/Agent as blocked-by-default on ad-supported pages, Cloudflare is making a bet that most site owners benefit from search visibility more than they risk from training bots. The September 15, 2026 date is notable because it will affect major multi-purpose crawlers like Googlebot, Applebot, and BingBot—bots that crawl for both search and training—forcing those operators to either separate their crawlers (as Cloudflare encourages) or accept blocking on sites that opt for the restrictive defaults.