On October 30, four companies that compete on nearly everything agreed on one line of text. OpenAI, Google, Microsoft, and Anthropic said they will honor a shared opt-out token: a single policy signal inside a website's robots.txt file that tells every AI crawler at once not to use the site's content for model training. Anthropic has already implemented it. Google and Microsoft plan to follow in 2027. OpenAI says it will honor the signal in the coming months.
The mechanism is deliberately boring. robots.txt has directed crawlers since 1994 — search engines, archivers, scrapers. The new token adds one more instruction to a file every web administrator already knows how to edit: this content is not for training. And it separates two things publishers have been forced to bundle: appearing in search results and feeding a training run. A site can stay findable and still say no to being ingested.
That separation is the quiet victory here, and it is worth pausing on, because the fight over AI training data has mostly been fought in courtrooms and press releases. Publishers sued. Labs argued fair use. Regulators in Brussels wrote the EU AI Act, whose code of practice for general-purpose models nudged this agreement along. All of that is still happening. But the thing that actually shipped this week is not a verdict or a law. It is a shared signal — four competitors adopting the same mechanism so that a publisher's "no" only has to be said once.
This is what coordination looks like when it works. The hard part of the training-data fight was never the technology of opting out. Any single lab could have honored a token years ago, and several built their own incompatible versions. The hard part was the agreement: one token, honored by all of them, so that a small publisher doesn't need a legal department and four different settings pages to exercise a choice. The cost of saying no fell from a negotiation to a line of text. That is infrastructure, not policy — and infrastructure is what makes a right usable.
There is a catch, and it is the interesting one. A token is a promise, not a fence. Nothing in robots.txt physically stops a crawler; the file has always been a request, not a lock. The signal works because the four labs said they will honor it, and because the EU's rules give that promise weight. But a crawl that doesn't happen leaves no trace. A publisher can verify that its content was used — it shows up in the model, in the outputs, in the regurgitated paragraph. It cannot verify that a crawler stayed away. The opt-out is auditable only in its violation, never in its observance.
That asymmetry is the next problem, and it is a design problem rather than a legal one. If consent is going to be machine-readable, compliance has to become machine-checkable. The honest version of this system has crawler transparency logs, published fetch records, a way for a publisher to ask "did you come here?" and get an answer that isn't "trust us." Cloudflare already blocks AI bots by default for its customers; Really Simple Licensing lets publishers put a price on scraping instead of a wall around it. The token is the first shared word in this negotiation. The vocabulary for proving you kept your word is still missing.
It is also worth asking who writes the next line. The token was designed with publishers, civil society, and regulators at the table, and that is why it has a chance. But the history of the web's commons is a history of shared signals being captured by their biggest users: the search engines that defined robots.txt's de facto rules, the platforms that decided what "public" means. A standard adopted by four giants is a standard four giants can also revise. The safeguard is the same as the achievement — that the signal is shared, public, and simple enough that a fifth party, or a fiftieth, can adopt it without asking permission.
The web spent thirty years learning that its commons needs fences. This week it learned something slightly different: that the cheapest fence is one everyone agrees to see. One line of text, four signatures, and a publisher's "no" that only has to be said once. The question now is whether the same trick works for the harder cases — the crawler that ignores the token, the model trained before the token existed, the "no" that arrives after the data is already inside. A signal is only as strong as what happens when someone ignores it. The next line in robots.txt will have to answer that.
Loading…