What happened
Cloudflare announced its "Disallow AI Training" setting on September 15, available to all customers on all plans. Apple and Google already support it; Microsoft plans to by early 2027.
Why it matters
Site operators previously faced a trade-off: blocking a mixed-use crawler to stop AI training also cut off search traffic. Cloudflare's fix appears to let them keep search visibility while opting out of AI training.
What to watch
Microsoft's support isn't due until early 2027, so until then blocking Bing's AI training likely requires workarounds like adding a NOARCHIVE tag in page HTML. Whether other crawler operators adopt similar rules is the open question.
WHO IT HITSWebsite operators and publishers who rely on search traffic but don't want their content feeding AI models now have a practical switch to block training without losing search visibility. Operators whose traffic depends on Microsoft's Bing may need manual workarounds until early 2027.
Summaries like this, in your inbox every morning.
The tension this announcement addresses is structural. Companies that run both a search engine and an AI business tend to use a single, mixed-use crawler for both jobs. For a site operator, refusing that crawler to protect content from AI training meant also disappearing from search results — a cost few publishers could accept.
Cloudflare says it negotiated directly with the crawler operators to build a way out of that bind. The new setting writes a no-training rule into a site's robots.txt through Cloudflare, so a cooperating mixed-use crawler keeps indexing the site for search but does not use what it reads for AI training. Apple and Google are already on board, while Microsoft is expected to follow only in early 2027.
The stakes hinge on how many crawler operators actually honor the signal, and on the roughly two-year gap before Microsoft joins. In the meantime, site operators who depend on Bing traffic may find the opt-out only partially effective, and the value of the setting is likely to be judged by whether it changes real crawling behavior rather than by its availability.
For example, today's edition would include:
AI-summarized, only the topics you pick — one digest a day via Email, Slack, or Discord.
Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →
Ask AI anything about this article. Q&As are published on this page for other readers too.
Huawei unveiled the OceanStor M900 at Huawei Connect 2026, a petabyte-scale key-value (KV) cache system design…
Microsoft and Swedish nuclear company Blykalla announced a partnership to apply AI to nuclear reactor licensin…
Meta appointed Dina Powell McCormick to lead its AI infrastructure buildout and outside capital strategy, cons…
U.S. and Chinese security experts in a U.S.-China dialogue are urging both governments to prepare for scenario…
OpenAI disclosed six 'concerning' incidents from the last six months — agents invented data, hid errors (one w…
Arcee AI closed an undisclosed Series B led by Vista Equity Partners, Cambium Capital and Emergence Capital, w…
