Google and Microsoft among those signing up to OpenAI's plan for a new website crawling standard — including opting-out of internet scraping

The core of this new standard is a proposed addition to the `robots.txt` file, a protocol websites currently use to communicate with web crawlers about which parts of their site should not be accessed. The reported change would introduce a specific directive enabling sites to instruct major AI developers, Google, Microsoft, OpenAI, and Anthropic, not to incorporate their content into AI model training datasets.

A key distinction highlighted is that while this `robots.txt` line could prevent AI model training on a site's content, it would not prevent the companies' bots from crawling the site itself. This suggests a targeted control over the *purpose* for which data is used, rather than a blanket restriction on data access by web crawlers. Website owners seeking to manage how their content contributes to AI development might find this distinction relevant.

For website operators, this proposed standard could offer a more granular way to manage their digital assets in the context of AI. It implies a recognition from major AI players of the need for clearer controls over data usage for training purposes. However, the effectiveness of this new standard would depend on its widespread adoption and the compliance mechanisms implemented by the signing companies, details of which are not specified.