🔍 Read the full analysis: Does This Shared Opt-Out Keep Your Data Out Of AI Training? on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
OpenAI, Google, Microsoft and Anthropic have said they will honor a shared robots.txt signal asking them not to use a website’s content to train generative AI models. The signal is voluntary, does not block crawling or AI answers based on live web content, and will take effect on different schedules; Meta has voiced support without committing to honor it.
OpenAI, Google, Microsoft and Anthropic said on October 30 they will honor a shared robots.txt signal asking that website content not be used to train generative AI models. The move could let publishers express the preference with one rule instead of maintaining separate instructions for each company, but the signal is voluntary and each company has set a different implementation timetable.
The proposed rule uses the user-agent name __usage_ai-training with Disallow: /. It appears in an individual Internet-Draft called “Machine-Readable Training Preference Signal,” published by Anthropic standards lead Kevin Kelley. The draft is associated with the Internet Engineering Task Force’s AI Preferences working group, but it is not an adopted internet standard.
Anthropic says it has started implementing support, while noting that adding support across its services will take time. OpenAI says it aims to implement the signal “in the coming months.” Google says it aims to support it in 2027, with its crawler documentation specifying early 2027. Microsoft says it will honor the shared training preference from February 1, 2027, as it crawls websites.
Microsoft also announced a separate page-level preference for “AI Grounding,” covering website content used to inform answers in Bing, Copilot and Microsoft 365. Its changes take effect February 1, 2027. Microsoft says its existing NoArchive and NoCache preferences will stop counting as training or grounding preferences after March 1, 2027. It says the choices do not guarantee inclusion, traffic or payment.
Infographic 01 · Who backs it, and when
One line in robots.txt, four companies, four timelines.
A shared signal against AI training. Only Anthropic says it has started; the others name dates or months. The standard itself is still a draft.
- 30 Oct 2026Draft published, four companies announce supportdoneIndividual Internet-Draft by Kevin Kelley (Anthropic)
- 30 Oct 2026Anthropic: “has already started implementing support”startedIts standards lead on the IETF list: “It’ll take us some time to add support across our services.”
- Coming monthsOpenAI aims to implement itplanExisting GPTBot and OAI-SearchBot rules keep applying
- 16 + 19 Nov 2026Working group meets at IETF 127, San FranciscoopenOn 21 Oct a chair said no consensus has been declared yet
- 1 Feb 2027Microsoft honors the new preferences as it crawlsplanTraining in robots.txt or per page; grounding per page
- 1 Mar 2027Bing stops reading NoArchive / NoCache as AI preferencesdeadlinePublish the new settings before this date
- 2027Google plans support; its docs say “early 2027”planA “no” maps to Google-Extended unless you set that token yourself
- 3 May 2027Draft version 00 expires unless updatedInternet-Drafts are valid for at most six months
Binding? No, voluntary
- OpenAI calls it a “voluntary effort”.
- The vocabulary draft does not “ensure that preferences are followed”.
- The draft “has no formal standing in the IETF standards process”.
- Private agreements may override it, the draft says.
Who else?
- Meta supports a common signal, but did not say whether it will honor it.
- Apple, Amazon, xAI, Perplexity: no statement in the sources reviewed for this article.
- The draft’s acknowledgments list contributors “for identification only”, not supporters.
Sources: IETF draft, 30 Oct 2026 · Anthropic on the IETF list · AIPREF minutes, 21 Oct 2026 · Google, 30 Oct 2026 · Google crawler docs · OpenAI, 30 Oct 2026 · Microsoft Bing, Oct 2026 · Meta (Andy Stone on X). Company plans are company statements.
Powered by Thorsten Meyer AI
One Rule, Limited Control
For publishers, the shared signal could reduce the work of communicating a preference to several AI companies. A site that currently maintains separate crawler rules may be able to add one common instruction for training use, while keeping existing company-specific rules in place during the rollout. That may make a publisher’s stated preference easier to express consistently.
But the rule is not a technical lock. It communicates a request, and the announced support is voluntary. It does not itself prevent a crawler from accessing a site, remove material from models that have already been trained, or cover every way content can appear in AI products. In particular, training and AI answers are distinct uses: the shared signal concerns training, while Microsoft’s grounding preference is a separate control for content used in answers.
The distinction matters to anyone deciding what to allow. A publisher may want to permit search indexing while declining training, or restrict training without opting out of live-content use in AI answers. The announcements do not establish a universal setting for all those choices. The signal also creates no right to payment, and Microsoft says its preferences do not change eligibility for Bing search results. Google says Google-Extended does not affect Google Search.
Products Worth Considering
Bose TV Speaker Compact Soundbar with Bluetooth and HDMI-ARC, Black
Apple Watch Series 12 GPS Smartwatch, 46mm
Bose QuietComfort True Wireless Earbuds with Active Noise Cancellation, Black
As an affiliate, we earn on qualifying purchases.
How Company Rules Fit Together
Before this proposal, website operators often used separate instructions for different services. Examples include GPTBot for OpenAI, Google-Extended for Google and ClaudeBot for Anthropic. Microsoft has also used page-level NoArchive and NoCache tags. The new shared preference is intended to give participating companies a common signal for one specific purpose: generative AI model training.
Companies have described different relationships between the shared rule and their existing controls. Google says that when a site uses the new token and has no Google-Extended rule, it plans to interpret the shared instruction as if the publisher had disallowed Google-Extended. OpenAI says its existing crawler controls will continue to apply. Microsoft has specified when its older tags stop counting for training and grounding preferences. Anthropic’s crawler help page, as of October 31, still described its existing per-crawler controls.
The proposal’s status remains preliminary. The source report says the draft was published on October 30, and that at an AI Preferences working-group meeting on October 21, a co-chair said the group had not declared consensus on anything. The companies’ announcements therefore describe intended or current implementation choices, not a finalized IETF standard binding every crawler.
““has already started implementing support for this draft””
— Anthropic
Adoption and Coverage Remain Open
Meta’s implementation plans are unconfirmed. Its spokesperson endorsed publishers’ ability to express preferences but did not say whether Meta will honor this signal. The source report also found no statements from Apple, Amazon, xAI or Perplexity about adopting it. Support from the four announcing companies does not establish that other AI services will follow.
The rollout dates are not identical, and details may change as companies implement the draft. Anthropic says support has started but has not given a date for coverage across its services. OpenAI has offered a general target of the coming months. Google’s stated aim is 2027, while Microsoft has set a specific date. The draft’s eventual status in the IETF process is also unresolved.
It is also not yet clear how uniformly the rule will be interpreted or monitored across participating services. The signal is a machine-readable preference rather than an enforcement mechanism. It does not address content already used in training, and the shared training instruction alone does not specify whether a company may use crawled material to produce answers based on live web content.
Watch for Rollout Details
Website operators can review the draft and decide separately what they want to permit for training, AI answers and search. Those are different uses, and the announcements describe different controls for them. Sites adding the shared training rule may also want to retain existing company-specific crawler instructions until each provider has implemented the common signal and clarified how its rules interact.
Microsoft’s dates call for particular attention from sites that rely on NoArchive or NoCache: its announcement says the new training and grounding preferences take effect on February 1, 2027, and the older tags stop counting for those purposes after March 1, 2027. Publishers should check Microsoft’s documentation before those dates and confirm which page-level preferences match their needs.
Further developments to watch include implementation updates from each company, any commitment from Meta or other AI providers, and progress in the IETF working group. Until the draft’s status and adoption become clearer, the shared signal should be treated as an additional preference for participating crawlers—not a universal block on AI use of published material.
Key Questions
What does the shared AI training opt-out do?
It is a proposed robots.txt rule expressing that a publisher does not want its content used to train generative AI models. OpenAI, Google, Microsoft and Anthropic have said they will honor it on differing schedules.
Does the rule stop AI companies from crawling a website?
No. The shared signal communicates a preference about training use; it is not a technical block on crawling. The source material describes it as a request, not a lock.
Does opting out of training also remove a site from AI answers?
Not under the shared training signal alone. The signal does not cover AI answers based on live web content. Microsoft announced a separate AI Grounding preference for content used in answers through Bing, Copilot and Microsoft 365.
Will Meta honor the shared signal?
That has not been confirmed. Meta spokesperson Andy Stone expressed support for publishers’ ability to state preferences, but did not say whether Meta will implement or honor the signal.
Is the shared rule an official internet standard?
No. It is an individual Internet-Draft linked to the IETF’s AI Preferences working group. The source report says the group had not declared consensus, so the draft does not yet have formal standing as an adopted IETF standard.
Source: ThorstenMeyerAI.com
