OpenAI to honor shared robots.txt AI training signal

The change targets a practical headache for anyone running a website: under the previous setup, operators had to identify individual AI crawlers and maintain separate rules for each and every provider. A shared signal means one preference can reach multiple participating providers at once, reducing that workload.

OpenAI frames the move as building on work already under way at the Internet Engineering Task Force's AI Preferences working group, where it says it is joining other participants in today's announcement. The company says this strengthens its continued participation in IETF standards discussions and that it will keep adapting its practices as those conversations evolve.

Importantly, the announcement is a commitment rather than a live feature: OpenAI says it aims to implement the signal in the coming months and will share details at that point. It also stresses that the new signal sits alongside, rather than replaces, the existing crawler controls it already offers through robots.txt. OpenAI encourages other AI providers to honor the signal, though whether they do is not stated.