OpenAI, Google, Microsoft back shared __usage_ai-training robots.txt signal

Until now, a site that wanted to keep its pages out of AI training had to identify each company's crawler and write a separate rule for it. The draft from Anthropic's Kevin Kelly, published as an Internet-Draft with the IETF, replaces that with one policy token, __usage_ai-training, placed under User-Agent in robots.txt. Because it introduces no new directives or grammar, existing parsers should keep working, and sites that never add the token are read as expressing no particular preference.

The companies are moving on different schedules. OpenAI aims to support the signal within the next few months and says it will publish details at implementation, while Google is aiming for 2027 and will treat a common-signal refusal as also covering Google-Extended unless a site explicitly opts in. Microsoft will start reading and applying the settings from February 1, 2027, and adds page-level meta tag and HTTP header controls that take priority over robots.txt, including a DisallowExtended option that still allows use in Copilot and Microsoft 365 but blocks grounding through Web IQ and Microsoft Foundry APIs.

The draft is filed in a personal capacity and has no formal standing in the IETF process yet, so the commitments are voluntary. The backdrop is a run of publisher lawsuits over AI use of web content, alongside earlier controls such as Cloudflare's July release of crawler management by search, agent and training purpose, and Google's June announcement of a Search Console toggle to opt out of AI search features.