AI giants back shared 'policy token' opt-out in robots.txt

The announcement lands as publishers contend with a continuing influx of AI crawlers scraping their sites and feeding content into AI models. Some organizations have already tried other routes: the Really Simple Licensing (RSL) standard lets publishers charge crawlers that visit their sites, and Cloudflare blocks AI crawlers by default. The shared token is a different approach, aiming to replace per-company preference settings with one signal in robots.txt that multiple providers read. The four companies say they will restrict their AI crawlers from accessing a site if they encounter the shared token. Meta communications head Andy Stone signaled support for the new setting but did not say whether the company will honor it, saying only that a common signal conveying preferences to multiple providers at once is an important step. Microsoft adds that it will offer granular controls letting publishers opt out of AI training or AI grounding, the process of pointing a model to a verified source to reduce hallucinations, while noting sites will still appear in Bing search results. Google previously implemented similar controls without impacting search visibility. Timing differs: Anthropic has already adopted the signal, Google and Microsoft plan to support it next year, and OpenAI says it will implement it in the "coming months."