Google, OpenAI back single robots.txt AI opt-out

The announcement builds on robots.txt, the long-standing text file many websites use to tell crawlers what they may access. Historically that file managed search engines; it now also helps regulate scrapers that AI providers use to collect training data. The friction is practical: a company wanting to disallow training-related scraping must add a separate entry for every AI provider, and each scraper needs its own entry if a provider uses multiple.

The four companies plan to fold their standard into the Robots Exclusion Protocol, the standardized format robots.txt follows, and have already submitted a draft to the Internet Engineering Task Force, the nonprofit that maintains it. The setting would not apply to models that perform exclusively non-generative tasks. Microsoft also plans a separate AI Grounding setting for requests made to answer a user question, and OpenAI plans similar controls. Google, Microsoft, Anthropic and OpenAI are backing the effort, while Meta has voiced support for the idea without committing to implement it.