
What happened
On October 30, OpenAI, Google, Microsoft and Anthropic said they will honor the __usage_ai-training robots.txt token, which a site writes with a User-Agent and Disallow: / line to opt out.
Why it matters
The token lets publishers express AI training preferences to four major providers in a single step, likely sparing them from writing separate rules per provider.
WHO IT HITSPublishers, news organizations and site operators who rely on robots.txt to control how their content is used for AI training could set one shared rule across four major AI labs instead of negotiating each provider separately.
Summaries like this, every morning.
The announcement unites four labs that have often competed over how web content flows into AI systems. Behind the token sits an individual IETF Internet-Draft, "Machine-Readable Training Preference Signal", dated October 30, 2026, expiring May 3, 2027, with the AIPREF working group chaired by Cloudflare's Mark Nottingham and Cisco's Suresh Krishnan. The participant list — Meta, Cloudflare, the Common Crawl Foundation, News Corp, the Financial Times, the BBC, the Electronic Frontier Foundation and Creative Commons — shows the discussion extends well beyond the four labs.
Google's John Mueller framed the step as helping a site state AI training preferences across multiple providers at once, and Google plans to read an opt-out via the token as if the publisher had disallowed Google-Extended. Microsoft, for its part, intends to build on the shared preference while adding a separate setting for AI grounding, the retrieval step that pulls live content into chat answers rather than training weights.
Two limits remain. OpenAI describes the mechanism as a coordination tool rather than a security control under RFC 9309, so non-compliant scrapers can ignore it, and the opt-out does not pull material already gathered into datasets or baked into models. The timelines also differ sharply: Anthropic says it has begun implementation, OpenAI speaks of the coming months, and Microsoft and Google point to 2027.
Pick your industry and the AI tools you use, and get news related to your work every day.
Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →
Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.
Uravation's free AI Checker recorded 121,608 judgments from July 18 to October 16, 2026; October's first half…

Google, Microsoft, Anthropic and OpenAI said they will support a new crawling standard that blocks all generat…
Tesla CEO Elon Musk warned that more than 200 AI neocloud providers could face large-scale failures or consoli…

Anthropic renamed "Max 5x" to "Max" and "Max 20x" to "Ultra" at unchanged monthly prices of $100 and $200, rem…

IPA released the Data & AI category Syllabus (Draft) Ver.0.1, which covers Subject A-2's eight mid-level knowl…

Dulo Inc., co-founded by Sebastian Thrun and Adrian Aoun, raised $130 million, with backers including former G…