Can you stop AI companies training on your website?
Four AI companies now back one robots.txt line for opting out of training. Here is what it does, what it does not do yet, and copy-paste rules for a small business site.
By Shashi Kumar ·

For years the honest answer to "can I stop AI companies training on my website?" was: sort of, one company at a time. You had to learn a different bot name for every lab, and hope each one kept its promise.
That changed a little on 30 October 2026. Google, Microsoft, OpenAI and Anthropic each said they will honour one shared line in robots.txt that means "do not train AI on this". It is a real step forward. It is also not switched on everywhere yet, and it does nothing about whether your site shows up in AI search answers, which for many small businesses is the part that actually matters.
What was announced, exactly
The new signal is a policy token called __usage_ai-training. It is defined in an IETF Internet-Draft, Machine-Readable Training Preference Signal, written by Kevin Kelley of Anthropic and published on 30 October 2026. You use it like any other robots.txt group:
User-Agent: __usage_ai-training
Disallow: /
A few details from the draft are worth knowing:
- It is an individual draft. The IETF itself says it has no formal standing yet, and it expires on 3 May 2027 unless revised.
- A
User-agent: *group never counts as this token. If you want to opt out of training, you have to name it. - It is about use, not access. The draft says the token "is not a directive for crawling behavior". Bots still visit; the token says what they may do with what they read.
Who has committed, and when
As of 9 November 2026, nobody has flipped this on across the board. The timelines differ:
| Company | What they said | When |
|---|---|---|
| Anthropic | Has started implementing it; rollout is gradual | From 30 Oct 2026 |
| OpenAI | Will honour it; current robots.txt rules stay in force meanwhile | "Next few months" |
| Microsoft (Bing) | Training opt-out via the token, plus new page-level grounding controls | From 1 Feb 2027 |
| Will treat the token as a Google-Extended opt-out | Early 2027 |
Sources: Google, Microsoft, Anthropic (below) and Search Engine Journal for OpenAI's timing.
Two details shape how you write your file. Anthropic says a rule that names ClaudeBot takes priority over the shared token. Google says that if you use the token and do not mention Google-Extended, it will act as if you had disallowed Google-Extended. So the token is the default, a vendor-specific rule overrides it, and you should keep your existing bot rules for now.
robots.txt in two minutes
robots.txt is a plain text file at yoursite.com/robots.txt that bots read before crawling. Three things trip people up:
- It is a request, not a lock. Good crawlers follow it; bad ones ignore it. Even Cloudflare's docs say compliance is voluntary.
- *A named group replaces the `
group.** If you write aUser-agent: GPTBot` section, GPTBot follows only that section and ignores your general rules. Repeat anything important. - Every subdomain needs its own file.
shop.yoursite.comdoes not inherit fromyoursite.com.
The crawler names that matter
Each name below was checked on the company's own page on 9 November 2026. Most companies now run separate bots for training and for search.
- OpenAI:
GPTBotis for training.OAI-SearchBotputs sites into ChatGPT search, and OpenAI says opted-out sites "will not be shown in ChatGPT search answers".ChatGPT-Userfetches a page when a person asks, and OpenAI says robots.txt rules may not apply to it. (OpenAI) - Anthropic:
ClaudeBotcollects training data.Claude-SearchBotindexes for search.Claude-Userfetches pages for a user's question. (Anthropic) - Google:
Google-Extendedis a control token, not a separate crawler. It covers Gemini training and grounding in Gemini Apps, and Google says it does not affect inclusion or ranking in Google Search. AI Overviews are part of Search, governed by Googlebot. (Google) - Perplexity:
PerplexityBotis for Perplexity's search results and, per Perplexity, not for training foundation models.Perplexity-Userfetches on request and "generally ignores robots.txt rules". (Perplexity) - Apple:
Applebot-Extendeddoes not crawl; it tells Apple not to train on what Applebot collected. Pages that disallow it can still appear in Siri and Spotlight results. (Apple) - Meta:
meta-externalagentcrawls for AI training.meta-webindexerhelps Meta AI cite content.meta-externalfetcherhandles user requests and may bypass robots.txt. (Meta) - Common Crawl:
CCBotbuilds Common Crawl's open web archive, which anyone, AI developers included, can download and analyse. (Common Crawl)
The real decision: training vs AI search
Blocking training and disappearing from AI answers are two different choices. Mixing them up is an easy mistake, and an expensive one.
Think of a CA firm in Ranchi with clear GST explainers. When someone asks ChatGPT or Perplexity "who does GST filing near me", the answer is built from pages the search bots could read. Block those bots and you remove yourself from the shortlist. Block only the training bots and you still appear, while saying no to being folded into the next model.
| Allow everything | Block training only | Block training and AI search | |
|---|---|---|---|
| Google Search and AI Overviews | Yes | Yes | Yes |
| Cited in ChatGPT, Claude, Perplexity search | Yes | Yes | Mostly no (user-triggered fetches may still happen) |
| Gemini app grounding | Yes | No, if you block Google-Extended | No |
| Used to train future models | Possibly | Not by bots that honour the rules | Not by bots that honour the rules |
| Best for | Local businesses chasing reach | Most businesses with original content | Paid, premium or sensitive content |
As of 9 Nov 2026. Vendor behaviour and the shared token's rollout are changing; check each company's crawler page.
Copy-paste robots.txt examples
Swap in your own domain, then open yoursite.com/robots.txt to check it.
1. Block training, stay visible in AI search (our default suggestion)
# Shared AI-training opt-out (draft; support rolling out 2026-2027)
User-Agent: __usage_ai-training
Disallow: /
# Vendor training controls that work today
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: meta-externalagent
Disallow: /
User-agent: CCBot
Disallow: /
# Everyone else, including search and AI-search bots
User-agent: *
Disallow: /admin/
Sitemap: https://example.com/sitemap.xml
If Gemini app visibility matters more to you than Gemini training, drop the Google-Extended group; it covers both.
2. Block training and AI search, keep Google Search
User-Agent: __usage_ai-training
Disallow: /
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: meta-webindexer
User-agent: CCBot
Disallow: /
User-agent: *
Disallow: /admin/
Sitemap: https://example.com/sitemap.xml
Stacking several User-agent lines over one rule set is valid. Some user-triggered fetchers say they may not follow it.
3. Opt out of training except for one folder
This is the pattern from the draft itself: say no to training everywhere except your press kit or public guides.
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
Beyond robots.txt: Cloudflare-style controls
If your site sits behind Cloudflare, you get two extra levers. Its managed robots.txt setting can write AI rules for you, including a Content-signal line with search, ai-input and ai-train preferences, and Cloudflare says it is available on all plans. Its AI Crawl Control tools can then actually block named AI bots at the network edge, instead of politely asking. A sign on the door versus a guard at the door. Other CDNs and hosts offer similar rules.
What about llms.txt?
llms.txt is a Markdown file at your site root that gives AI tools a short summary of your site and links to its most useful pages. Its own site calls it "a proposal". It is not an IETF standard, none of the opt-out controls above depend on it, and it does not block anything. It is cheap to add if you want AI tools to understand your services, but it is a signpost, not a fence.
What we know vs what we don't
We know: the token's syntax, that four major labs have publicly backed it, and the dates Microsoft and Google have given.
We don't know: when OpenAI's support goes live, whether smaller crawlers will adopt it, and whether the draft becomes a formal standard. Nothing here removes content already used for training.
Frequently asked questions
Does the new shared AI-training opt-out work today?
Partly. Anthropic says it has begun rolling it out, OpenAI says it will follow within months, Microsoft starts on 1 February 2027 and Google says early 2027. Until then, keep vendor rules like GPTBot and Google-Extended in place.
Will blocking GPTBot remove my site from ChatGPT search?
No. GPTBot is OpenAI's training crawler. ChatGPT search uses OAI-SearchBot, and only blocking that bot removes you from ChatGPT search answers.
Does blocking Google-Extended hurt my Google ranking?
Google says Google-Extended does not affect inclusion in Google Search and is not a ranking signal. It does, however, cover grounding in the Gemini app, so blocking it can reduce your visibility there.
Can robots.txt remove my content from models already trained?
No. robots.txt and the new token only express preferences for future crawling and training. They do not reach back into existing datasets or models.
Is llms.txt a standard I need?
No. It is a community proposal that helps AI tools find a summary of your site. It is optional and blocks nothing.
The bottom line
The shared token is good news: one line instead of a dozen, backed by the four biggest names. But for the next few months the safe setup is belt and braces, the token plus the vendor rules, with search bots left open unless you have a reason to hide.
If you are still deciding how AI should find your business, start with the basics in our SEO guide for small business websites, check that you actually control your domain and hosting, and see which assistants your customers use in ChatGPT vs Gemini vs Claude. If you would rather have someone set up your robots.txt, bot rules and an AI-friendly site structure properly, talk to us.
Sources and further reading
- Machine-Readable Training Preference Signal (draft-kelley-aipref-training-preference-signal-00) · IETF Datatracker · 2026-10-30
- Supporting a shared AI training control for websites · Google Search Central Blog · 2026-10-30
- Your content, your choice: clearer preferences for training and grounding · Microsoft Bing Blogs · 2026-10-30
- Google, Microsoft & OpenAI Plan To Honor Shared AI Training Opt-Out · Search Engine Journal · 2026-11-01
- Overview of OpenAI Crawlers · OpenAI · 2026-11-09
- Does Anthropic crawl data from the web, and how can site owners block the crawler? · Claude Help Center · 2026-11-09
- Google's common crawlers · Google for Developers · 2026-11-09
- Perplexity Crawlers · Perplexity · 2026-11-09
- About Applebot · Apple Support · 2026-11-09
- Meta Web Crawlers · Meta for Developers · 2026-11-09
- Common Crawl - CCBot · Common Crawl · 2026-11-09
- robots.txt setting · Cloudflare Docs · 2026-11-09
- The /llms.txt file, v2 · llmstxt.org · 2026-08-10
Contact Acquiretech Software Solution: info@acquiretechsoftwaresolution.in · WhatsApp +91 80786 33912 · Send an enquiry