Cloudflare is replacing its blanket AI crawler block with category-based controls, allowing site owners to separately manage search, training, and agent bots. Starting September 15, 2026, training and agent bots will be blocked by default on ad-supported pages, while search crawlers stay allowed. The change responds to CEO Matthew Prince's longstanding criticism of Google for bundling search and AI crawlers together, and reflects his June 2026 observation that bot traffic exceeded human traffic on the internet for the first time.
Summaries like this, in your inbox every morning.
Sign up free →What happened
Cloudflare replaced its blanket AI bot blocker with granular controls that let site owners manage search crawlers, training crawlers, and agent bots separately. Starting September 15, 2026, training and agent bots will be blocked by default on pages carrying ads, while search crawlers remain allowed. Enterprise customers now have access to BotBase, a searchable database showing how each bot is classified and uses content.
Why it matters
The shift reflects tension between site owners and AI companies over data access. Cloudflare CEO Matthew Prince has publicly criticized Google for bundling its search crawler with its AI training crawler, making it impossible for sites to allow one but not the other. In June 2026, Prince noted that bot traffic surpassed human traffic on the internet for the first time, a milestone he originally expected to occur in late 2027.
What to watch
Verified bots will no longer get automatic access; bot operators must now prove they identify themselves honestly and don't abuse their access. The new controls are available to all customers, including those on the free plan. Multi-purpose crawlers like Googlebot will be treated according to whichever rule is strictest.
Cloudflare's move reflects a growing friction between website operators and AI companies over data collection rights. Since July 2024, Cloudflare customers have had the ability to block all AI crawlers with a single click, but this blanket approach created a false choice: sites could either allow all AI bots or none at all. The new granular controls acknowledge that site owners may want different policies for different bot purposes—permitting search engine indexing (which drives visibility) while restricting bulk data collection for AI model training. The September 15, 2026 default blocking of training and agent bots on ad-supported pages is particularly significant: it signals that Cloudflare considers ad presence a reliable proxy for a site's preference for human traffic over AI data collection.
Cloudflare CEO Matthew Prince has been a vocal critic of Google's crawler bundling strategy, which forces sites to choose between losing search visibility and accepting AI training access. His June 2026 observation that bot traffic surpassed human traffic for the first time underscores the scale at which this issue now operates. By requiring bot operators to split their crawlers by purpose and to prove honest identification, Cloudflare is attempting to rebalance power: sites gain transparency and control, while bots must earn trust through compliance rather than relying on a "verified" label. The expansion of these controls to free-tier customers suggests Cloudflare aims to make granular bot management a standard expectation across the web.
No discussion yet for this article
Get curated AI news from 200+ sources delivered daily to your inbox. Free to use.
Get Started FreeFree · takes 30 seconds · unsubscribe anytime
1 minute a day. The AI essentials.
200+ sources · Email / LINE / Slack
