What happened
Cloudflare announced its "Disallow AI Training" setting on September 15, available to all customers on all plans. Apple and Google already support it; Microsoft plans to by early 2027.
Why it matters
Site operators previously faced a trade-off: blocking a mixed-use crawler to stop AI training also cut off search traffic. Cloudflare's fix appears to let them keep search visibility while opting out of AI training.
What to watch
Microsoft's support isn't due until early 2027, so until then blocking Bing's AI training likely requires workarounds like adding a NOARCHIVE tag in page HTML. Whether other crawler operators adopt similar rules is the open question.
WHO IT HITSWebsite operators and publishers who rely on search traffic but don't want their content feeding AI models now have a practical switch to block training without losing search visibility. Operators whose traffic depends on Microsoft's Bing may need manual workarounds until early 2027.
Summaries like this, every morning.
The tension this announcement addresses is structural. Companies that run both a search engine and an AI business tend to use a single, mixed-use crawler for both jobs. For a site operator, refusing that crawler to protect content from AI training meant also disappearing from search results — a cost few publishers could accept.
Cloudflare says it negotiated directly with the crawler operators to build a way out of that bind. The new setting writes a no-training rule into a site's robots.txt through Cloudflare, so a cooperating mixed-use crawler keeps indexing the site for search but does not use what it reads for AI training. Apple and Google are already on board, while Microsoft is expected to follow only in early 2027.
The stakes hinge on how many crawler operators actually honor the signal, and on the roughly two-year gap before Microsoft joins. In the meantime, site operators who depend on Bing traffic may find the opt-out only partially effective, and the value of the setting is likely to be judged by whether it changes real crawling behavior rather than by its availability.
Pick your industry and the AI tools you use, and get news related to your work every day.
Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →
Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.
A teacher-facing analysis treats AI-text detectors as production binary classifiers using proxy features (toke…
Discovered Materials announced a $1 million prize pool for solutions to five materials science problems — incl…
OpenAI published a National Community Compact and a Compute Campus Policy Blueprint
On October 21, 2026, Anfini CEO Jun Hashimoto gave a special lecture at Meiji University's engineering faculty…
On November 3, voters in 10 states backed restrictions on data center development in 40 of 41 local ballot mea…
A six-year web designer who signed up for Claude Pro in April 2026 and later moved to Max published the first…
