AI giants promise publishers new way to block some scraping

The proposed solution introduces a unified instruction that, once adopted, would be recognized by participating AI companies, streamlining the process for publishers to opt out of content scraping for AI training. This contrasts with the current fragmented approach, which has led to increased criticism and legal actions from publishers, including a lawsuit by The New York Times against OpenAI and Microsoft alleging unauthorized use of its articles.

However, the practical impact of this new standard will not be immediate. Microsoft plans to implement the solution starting February 1, 2027, with Google following "sometime in 2027." OpenAI expects to roll it out "in the coming months," while Anthropic states it has "already started." This staggered implementation means a widespread, uniform effect across the industry is still months away.

The scope of the new setting is also specific: it applies only to future training of AI models and does not prevent general web crawling. Content scraping will remain the default until a publisher actively changes their settings. Microsoft and Google also note that opting out could reduce the chances that a publisher's content will surface in AI applications and experiences, presenting a potential trade-off for content providers.