Training bots vs search bots: which is which
Each AI company runs separate bots for training its models and for finding pages to cite. Block the first kind and nothing changes for customers. Block the second and that assistant can't name you.
| Company | Training only | Search: blocking hides you | Fetches a page when a user asks |
|---|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot | ChatGPT-User |
| Anthropic | ClaudeBot | Claude-SearchBot | Claude-User |
| Perplexity | None listed | PerplexityBot | Perplexity-User |
| Google-Extended (also Gemini app answers) | Googlebot | ||
| Apple | Applebot-Extended | Applebot (Siri, Spotlight, Safari) | |
| Microsoft | No separate bot listed | Bingbot (Bing and Copilot) |
Sources: OpenAI, Anthropic, Perplexity, Google, Apple, Bing.
OpenAI spells out that the two are separate choices:
"Each setting is independent of the others – for example, a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training OpenAI's generative AI foundation models." (OpenAI)
And what blocking the search bot costs: sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers, though can still appear as navigational links" (OpenAI). Anthropic says the same of Claude-SearchBot: blocking it "may reduce your site's visibility and accuracy in user search results" (Anthropic).
Should I block GPTBot?
It's your call, and it won't cost you ChatGPT answers. GPTBot only collects pages "that may be used in training" (OpenAI). Apple is just as clear about its training token: "Webpages that disallow Applebot-Extended can still be included in search results" (Apple).
For most service businesses the stakes are low either way. A page that lists your services, areas and hours is meant to be repeated. Original articles or paid course material give you more reason to block.
The Google-Extended trap
Blocking Google-Extended keeps you in Google Search but takes you out of the Gemini app's grounded answers. Google uses the token for Gemini training and for grounding "in Gemini Apps" with content from its search index (Google).
"Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search." (Google)
Google's The AI-written summary Google shows above some search results, with links to the pages it drew from. More and AI Mode have their own switch. Since August 31, 2026, every site has a Search generative AI control in Google's free tool that shows which searches bring people to your site and flags pages Google can't index. More, under Settings. Excluding your site removes it from AI Overviews, AI Mode and AI features in Discover:
"You won't receive any traffic or impressions from these features." (Search Console Help)
Google says the switch "isn't used as a ranking or inclusion signal affecting other parts of Search" (Search Console Help). For a business that wants customers, leaving it on Include makes sense.
How to block AI bots in robots.txt
Add two lines per bot to the robots.txt file at the root of your site. To keep training bots out of the whole site:
| Line | Meaning |
|---|---|
| User-agent: GPTBot | The next rule is for OpenAI's training bot |
| Disallow: / | Don't crawl any page |
| User-agent: ClaudeBot | Anthropic's training bot |
| Disallow: / | Don't crawl any page |
| User-agent: Applebot-Extended | Apple's training token |
| Disallow: / | Don't use any page for training |
- Block one folder instead by putting a path after Disallow, such as /blog/. Google's own example blocks an archive folder this way (Google).
- Repeat on every subdomain. Anthropic asks for the rule "for every subdomain that you wish to opt out from" (Anthropic).
- Allow a day. OpenAI and Perplexity each say changes can take about 24 hours (OpenAI, Perplexity).
Never add Disallow rules for OAI-SearchBot, Claude-SearchBot, Claude-User, PerplexityBot, Googlebot, Bingbot or Applebot unless you want out of those answers.
robots.txt is not security: noindex vs disallow
Robots.txt is a request. Google: "it's up to the crawler to obey them" (Google Search Central). Two of the user-triggered fetchers say outright they may not obey: ChatGPT-User ("robots.txt rules may not apply") and Perplexity-User ("generally ignores robots.txt rules") (OpenAI, Perplexity).
"A robots.txt file tells search engine crawlers which URLs the crawler can access on your site. This is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google." (Google Search Central)
| You want to | Use | Why |
|---|---|---|
| Stop a bot crawling | robots.txt Disallow | A disallowed page can still be indexed if other sites link to it (Google) |
| Keep a page out of Google | noindex, and don't also block it in robots.txt | Google must crawl the page to see noindex (Google) |
| Keep files private | A password | robots.txt can't enforce anything (Google) |
| Hide a page from Google fast | Search Console Removals tool | Lasts about six months, so add noindex or delete the page too (Search Console Help) |
Cloudflare and other firewalls
Check your firewall before your robots.txt. A bot blocked at the firewall never reads your rules.
In July 2025 Cloudflare began asking every new domain at sign-up whether to allow AI crawlers, with blocking as the default (Cloudflare). Its documentation, updated November 2, 2026, now says new domains default to allow for search, agent and training bots (Cloudflare Docs). If your site was set up in between, check what it chose.
Look in Security, then Settings, then AI Crawler and Agent access policies. Two details matter (Cloudflare Docs):
- Settings are split into Search, Agent and Training. Keep Search allowed.
- "Training actions also apply to mixed-purpose crawlers used for both Search and Training." Blocking Training can block a search crawler too. Disallow (a robots.txt rule) is the softer choice.
Bing and Copilot: control it with meta tags
Bing lists no separate AI bot: Bingbot crawls for both Bing and Copilot. To limit Copilot without leaving Bing, use Bing's page-level tags:
"NOARCHIVE prevents content from being used in Copilot responses and grounding results." (Bing Webmaster Guidelines)
NOCACHE is the middle setting: Copilot may use only "the URL, title, and snippet" (Bing Webmaster Guidelines).
Example
A plumber whose host blocks "AI scrapers"
The firewall's AI setting blocks everything, so ChatGPT and Perplexity never cite the site. The fix: Search and Agent set to allow, Training set to Disallow. Customers' assistants can now read the service pages; training bots are still asked to stay out.
A therapist who writes original articles
She disallows GPTBot, ClaudeBot and Applebot-Extended. She leaves Google-Extended alone, so the Gemini app can still use her pages, and keeps every search bot allowed.
A law firm with a client document folder
The folder was "hidden" with a robots.txt Disallow, which only asks bots to stay away. The fix is a login on the folder, plus noindex on any page that was already indexed.
Different for you?
- You want more AI mentions, not fewer: how to rank in Perplexity, Gemini and Claude and getting ChatGPT to recommend you.
- Your site isn't on Google at all: website not showing up on Google.
- Something about your business needs to come off Google: removing information from Google.
- Someone is selling you "AI bot optimization": is GEO worth paying for?
- You'd rather have it set up and checked: SEO & GEO Agent.