AIToday
Large Language ModelsAI Regulation & PolicyAI WeeklyPublished: Oct 31, 2026, 13:00 JST

Big AI labs back __usage_ai-training opt-out token

Send on LINE
Big AI labs back __usage_ai-training opt-out token

3 Key Points

  1. What happened

    On October 30, OpenAI, Google, Microsoft and Anthropic said they will honor the __usage_ai-training robots.txt token, which a site writes with a User-Agent and Disallow: / line to opt out.

  2. Why it matters

    The token lets publishers express AI training preferences to four major providers in a single step, likely sparing them from writing separate rules per provider.

WHO IT HITSPublishers, news organizations and site operators who rely on robots.txt to control how their content is used for AI training could set one shared rule across four major AI labs instead of negotiating each provider separately.

Not sure about something? Ask the AI

Questions and answers are published on this page.

Summaries like this, every morning.

Context & Analysis

The announcement unites four labs that have often competed over how web content flows into AI systems. Behind the token sits an individual IETF Internet-Draft, "Machine-Readable Training Preference Signal", dated October 30, 2026, expiring May 3, 2027, with the AIPREF working group chaired by Cloudflare's Mark Nottingham and Cisco's Suresh Krishnan. The participant list — Meta, Cloudflare, the Common Crawl Foundation, News Corp, the Financial Times, the BBC, the Electronic Frontier Foundation and Creative Commons — shows the discussion extends well beyond the four labs.

Google's John Mueller framed the step as helping a site state AI training preferences across multiple providers at once, and Google plans to read an opt-out via the token as if the publisher had disallowed Google-Extended. Microsoft, for its part, intends to build on the shared preference while adding a separate setting for AI grounding, the retrieval step that pulls live content into chat answers rather than training weights.

Two limits remain. OpenAI describes the mechanism as a coordination tool rather than a security control under RFC 9309, so non-compliant scrapers can ignore it, and the opt-out does not pull material already gathered into datasets or baked into models. The timelines also differ sharply: Anthropic says it has begun implementation, OpenAI speaks of the coming months, and Microsoft and Google point to 2027.

FAQ
How does a publisher actually use the token?
A site writes User-Agent: __usage_ai-training followed by Disallow: /, and can carve out exceptions with Allow: lines.
Does this remove content already used in training?
No. Updating robots.txt does not remove material already collected into existing datasets or trained models, and the token controls training, not fetching — bots may still pull the page.
Is this a formal standard?
No. It is an individual IETF Internet-Draft titled "Machine-Readable Training Preference Signal", authored by Anthropic's Kevin Kelley, and the IETF datatracker notes it is not endorsed by the IETF and has no formal standing.
Send on LINE

AI news that matters for your work, delivered every morning.

Pick your industry and the AI tools you use, and get news related to your work every day.

Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →

Ask AI

Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.

Questions and answers are published on this page.

Related Articles

Next articleNvidia weighs backstop for Nscale's West Virginia AI site