Products SEO & GEO Agent Paid Ads Agent Website Agent Conversion Agent
How it works Pricing FAQ Blog Book a call

Should you block AI crawlers? GPTBot, ClaudeBot and robots.txt

Updated · 5 min read

Want help applying this to your business? Talk it through on a free 30-minute call.

Book a call

Training bots vs search bots: which is which

Each AI company runs separate bots for training its models and for finding pages to cite. Block the first kind and nothing changes for customers. Block the second and that assistant can't name you.

CompanyTraining onlySearch: blocking hides youFetches a page when a user asks
OpenAIGPTBotOAI-SearchBotChatGPT-User
AnthropicClaudeBotClaude-SearchBotClaude-User
PerplexityNone listedPerplexityBotPerplexity-User
GoogleGoogle-Extended (also Gemini app answers)Googlebot
AppleApplebot-ExtendedApplebot (Siri, Spotlight, Safari)
MicrosoftNo separate bot listedBingbot (Bing and Copilot)

Sources: OpenAI, Anthropic, Perplexity, Google, Apple, Bing.

OpenAI spells out that the two are separate choices:

"Each setting is independent of the others – for example, a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training OpenAI's generative AI foundation models." (OpenAI)

And what blocking the search bot costs: sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers, though can still appear as navigational links" (OpenAI). Anthropic says the same of Claude-SearchBot: blocking it "may reduce your site's visibility and accuracy in user search results" (Anthropic).

Should I block GPTBot?

It's your call, and it won't cost you ChatGPT answers. GPTBot only collects pages "that may be used in training" (OpenAI). Apple is just as clear about its training token: "Webpages that disallow Applebot-Extended can still be included in search results" (Apple).

For most service businesses the stakes are low either way. A page that lists your services, areas and hours is meant to be repeated. Original articles or paid course material give you more reason to block.

The Google-Extended trap

Blocking Google-Extended keeps you in Google Search but takes you out of the Gemini app's grounded answers. Google uses the token for Gemini training and for grounding "in Gemini Apps" with content from its search index (Google).

"Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search." (Google)

Google's The AI-written summary Google shows above some search results, with links to the pages it drew from. More and AI Mode have their own switch. Since August 31, 2026, every site has a Search generative AI control in Google's free tool that shows which searches bring people to your site and flags pages Google can't index. More, under Settings. Excluding your site removes it from AI Overviews, AI Mode and AI features in Discover:

"You won't receive any traffic or impressions from these features." (Search Console Help)

Google says the switch "isn't used as a ranking or inclusion signal affecting other parts of Search" (Search Console Help). For a business that wants customers, leaving it on Include makes sense.

How to block AI bots in robots.txt

Add two lines per bot to the robots.txt file at the root of your site. To keep training bots out of the whole site:

LineMeaning
User-agent: GPTBotThe next rule is for OpenAI's training bot
Disallow: /Don't crawl any page
User-agent: ClaudeBotAnthropic's training bot
Disallow: /Don't crawl any page
User-agent: Applebot-ExtendedApple's training token
Disallow: /Don't use any page for training

Never add Disallow rules for OAI-SearchBot, Claude-SearchBot, Claude-User, PerplexityBot, Googlebot, Bingbot or Applebot unless you want out of those answers.

robots.txt is not security: noindex vs disallow

Robots.txt is a request. Google: "it's up to the crawler to obey them" (Google Search Central). Two of the user-triggered fetchers say outright they may not obey: ChatGPT-User ("robots.txt rules may not apply") and Perplexity-User ("generally ignores robots.txt rules") (OpenAI, Perplexity).

"A robots.txt file tells search engine crawlers which URLs the crawler can access on your site. This is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google." (Google Search Central)
You want toUseWhy
Stop a bot crawlingrobots.txt DisallowA disallowed page can still be indexed if other sites link to it (Google)
Keep a page out of Googlenoindex, and don't also block it in robots.txtGoogle must crawl the page to see noindex (Google)
Keep files privateA passwordrobots.txt can't enforce anything (Google)
Hide a page from Google fastSearch Console Removals toolLasts about six months, so add noindex or delete the page too (Search Console Help)

Cloudflare and other firewalls

Check your firewall before your robots.txt. A bot blocked at the firewall never reads your rules.

In July 2025 Cloudflare began asking every new domain at sign-up whether to allow AI crawlers, with blocking as the default (Cloudflare). Its documentation, updated November 2, 2026, now says new domains default to allow for search, agent and training bots (Cloudflare Docs). If your site was set up in between, check what it chose.

Look in Security, then Settings, then AI Crawler and Agent access policies. Two details matter (Cloudflare Docs):

Bing and Copilot: control it with meta tags

Bing lists no separate AI bot: Bingbot crawls for both Bing and Copilot. To limit Copilot without leaving Bing, use Bing's page-level tags:

"NOARCHIVE prevents content from being used in Copilot responses and grounding results." (Bing Webmaster Guidelines)

NOCACHE is the middle setting: Copilot may use only "the URL, title, and snippet" (Bing Webmaster Guidelines).

Example

A plumber whose host blocks "AI scrapers"

The firewall's AI setting blocks everything, so ChatGPT and Perplexity never cite the site. The fix: Search and Agent set to allow, Training set to Disallow. Customers' assistants can now read the service pages; training bots are still asked to stay out.

A therapist who writes original articles

She disallows GPTBot, ClaudeBot and Applebot-Extended. She leaves Google-Extended alone, so the Gemini app can still use her pages, and keeps every search bot allowed.

A law firm with a client document folder

The folder was "hidden" with a robots.txt Disallow, which only asks bots to stay away. The fix is a login on the folder, plus noindex on any page that was already indexed.

Different for you?

Keep learning

NextWebsite not showing up on Google: the checks in orderCan't find your website on Google? Run the checks in order: a site: search, URL Inspection, noindex and platform settings, new-site timing, Maps vs search.

Ask us one question

One line is enough. We reply by email.

Sources

  1. OpenAI: Overview of OpenAI Crawlers
  2. Claude Help Center: Does Anthropic crawl data from the web, and how can site owners block the crawler?
  3. Perplexity Docs: Perplexity crawlers
  4. Google Crawling Infrastructure: Google's common crawlers (Google-Extended)
  5. Search Console Help: Search generative AI control
  6. Apple Support: About Applebot
  7. Bing Webmaster Guidelines
  8. Google Search Central: Introduction to robots.txt
  9. Google Search Central: Block Search indexing with noindex
  10. Search Console Help: Removals tool
  11. Cloudflare press release: permission-based approach for AI crawlers (July 2025)
  12. Cloudflare Docs: AI Crawler and Agent access policies