Products SEO & GEO Agent Paid Ads Agent Website Agent Conversion Agent
How it works Pricing FAQ Blog Book a call

AI crawler logs: see which AI bots visit your site

Updated · 8 min read

Short answer Download your raw access log from your host (or open Cloudflare's AI Crawl Control) and search for the bot names: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot, Applebot. Check the IP against each company's published list, since names can be faked. Google Analytics won't show them: it drops known bots.

Want help applying this to your business? Talk it through on a free 30-minute call.

Book a call

Where do I find my server logs?

With your web host, not in Google Analytics. A server log (also called an access log) is a text file where your server writes one line for every request, from people and bots alike. Analytics tools only count visitors who run their tracking code, and Google Analytics goes further:

"In Google Analytics properties, traffic from known bots and spiders is automatically excluded." (Analytics Help: Known bot-traffic exclusion)

Google adds that you "cannot disable" this or see how much was removed. So the log is the only place AI bots show up, unless a firewall in front of your site counts them for you.

Your setupWhere to look
Shared hosting with cPanelMetrics, then Raw Access. It downloads a compressed .gz file of your access log, for Apache or NGINX servers (cPanel Docs: Raw Access)
Your own server (VPS)The web server's access log file. Ask whoever set it up for the path
Site behind CloudflareAI Crawl Control. Cloudflare says it shows "which AI services are accessing your content" and is available on all plans (Cloudflare Docs: AI Crawl Control)
Wix, Squarespace or another hosted builderAsk the platform's support whether raw logs are available

Two cPanel details matter. Your host can switch the Raw Access screen off. And cPanel saves older logs only if you tick the archive box in that screen; another box deletes last month's archives at month end (cPanel Docs: Raw Access). Turn archiving on now if you want a few months to compare.

Which bot names to search for

Open the log in a text editor or spreadsheet and search for these names. Each company documents its bots, and what a visit means differs by bot.

Search forCompanyWhat a visit means
GPTBotOpenAICollecting pages that may train its models
OAI-SearchBotOpenAIIndexing pages for ChatGPT search answers
ChatGPT-UserOpenAIA ChatGPT user's question just led it to your page
OAI-AdsBotOpenAIChecking a page you submitted as a ChatGPT ad
ClaudeBotAnthropicCollecting training data
Claude-SearchBotAnthropicIndexing pages for Claude's search answers
Claude-UserAnthropicA Claude user's question led it to your page
PerplexityBotPerplexityIndexing pages for Perplexity answers
Perplexity-UserPerplexityA Perplexity user's question led it to your page
GooglebotGoogleGoogle Search; AI Overviews and AI Mode link only to pages in that index
ApplebotAppleSiri, Spotlight and Safari search
bingbotMicrosoftBing, whose index also feeds Copilot answers

Sources: OpenAI: Overview of OpenAI Crawlers; Claude Help Center; Perplexity Docs; Google Search Central: AI features and your website; Apple Support: About Applebot; Bing Webmaster Guidelines.

Don't look for Google-Extended or Applebot-Extended. They are settings you write in robots.txt, not visitors. Google says Google-Extended "doesn't have a separate HTTP request user agent string" (Google's common crawlers), and Apple says "Applebot-Extended does not crawl webpages" (Apple Support: About Applebot). Their traffic appears as Googlebot and Applebot.

The "-User" lines are the most telling for a business. OpenAI says ChatGPT-User visits a page "when users ask ChatGPT or a CustomGPT a question." A hit on your pricing page from one of these means someone asked an assistant something and it opened your page to answer. Whether to block any of these bots is a separate question: should you block AI crawlers?

How to read one log line

Most hosts use what Apache calls the Combined Log Format (Apache HTTP Server Docs: Log Files). Each line holds the same fields in the same order:

FieldExampleWhat it tells you
IP address203.0.113.7Who asked; use it to check the bot is real
Date and time[16/Nov/2026:09:14:02 -0500]When
Request"GET /furnace-repair/ HTTP/1.1"Which page
Status code200200 is fine; 301 redirected; 403 blocked; 404 not found; 5xx server error
Size18342Bytes sent back
Referrer"-"The page that linked here; bots usually send none
User agent"...compatible; GPTBot/1.4; +https://openai.com/gptbot"The name the visitor gives itself

The bot name sits inside the last field, the user agent. OpenAI notes the version number in it may change, so search for "GPTBot", not "GPTBot/1.4".

Is it really GPTBot? Check the IP address

Anyone can type "GPTBot" into a request, so a name alone proves nothing. Google's verification page exists for exactly this, in case "spammers or other troublemakers" are posing as Google (Verify requests from Google crawlers and fetchers). Check the line's IP address against the list the company publishes:

BotHow to check
OpenAI botsIP lists at openai.com/gptbot.json, searchbot.json and chatgpt-user.json (OpenAI)
Anthropic botsIP list at claude.com/crawling/bots.json. Anthropic: an IP on that list "indicates that the crawler is coming from Anthropic"
Perplexity botsIP lists at perplexity.ai/perplexitybot.json and perplexity-user.json (Perplexity Docs)
GooglebotReverse DNS lookup must end in googlebot.com, google.com or googleusercontent.com, and the forward lookup must return the same IP (Google)
ApplebotReverse DNS in applebot.apple.com, or the IP list on Apple's page (Apple Support)
BingbotIP list at bing.com/toolbox/bingbot.json

Perplexity's advice applies to every list: "Always use the most current IP ranges from the official JSON endpoints." The ranges change. For a handful of lines, paste the IP into a CIDR checker (a free tool that says whether an address falls inside a range) next to the list. Google says the manual method "is sufficient for most use cases."

What do AI bot visits tell you?

Mostly whether bots can reach you, not how well you rank. Google is blunt about its own crawler:

"Google uses many signals to rank the results, and while crawling is necessary for a page to be in search results, it's not a ranking signal." (Google: Myths about crawling)

So a page hit 40 times isn't "preferred" over one hit twice. No AI company says crawl counts reflect how much it values a page. Read the log for these problems instead:

What you seeWhat it likely meansWhat to do
No OAI-SearchBot, Claude-SearchBot or PerplexityBot lines at allSomething may block them, often a firewall or host bot settingCheck your firewall's AI bot settings and robots.txt
Search bots get 403Your server or firewall refuses themAllow search bots; OpenAI recommends "allowing requests from our published IP ranges"
Bots hit 404 on old URLsLinks point to pages that movedRedirect moved pages people still link to
5xx errorsYour server failedAsk your host; Google advises keeping server errors low
Main service pages never visitedBots may not find themLink them from the homepage and menu; check pages not indexed
One bot floods the serverToo many requests at onceClaudeBot honors a Crawl-delay line in robots.txt (Claude Help Center)

Do I need to worry about crawl budget?

Almost certainly not. Crawl budget is how many pages Google will crawl on a site in a given time. Google's guide is aimed at sites with "1 million+ unique pages" changing weekly, or 10,000+ changing daily, and says:

"If your site doesn't have a large number of pages that change rapidly, or if your pages seem to be crawled the same day that they are published, you don't need to read this guide." (Google: Optimize your crawl budget)

A 30-page service business site is far below that. For Googlebot, Google points owners to the Crawl Stats report in Search Console instead (Google: Myths about crawling); the Search Console guide covers the reports worth reading.

Can I upload my log to ChatGPT to analyze it?

You can, but strip the IP address column first. Every human visitor's IP is in that file, and you don't need it once the bots are verified. A spreadsheet works just as well for a small site: split the file into columns on spaces, filter the user agent column for each bot name, then count rows by page and status code.

Example

Illustrative cases. Numbers are invented for the example.

A roofer with no AI search bot visits

A month of logs shows Googlebot and GPTBot but no OAI-SearchBot or PerplexityBot. The host's "block AI scrapers" switch was on. After allowing search bots, OAI-SearchBot lines appear within a week, with status 200.

A dental clinic checking "GPTBot" traffic

Thousands of GPTBot lines a day slow the site. Half the IPs aren't on OpenAI's list, so they're fakes posing as GPTBot. The clinic blocks those IPs at the firewall and leaves the real ones alone.

An accountant who sees ChatGPT-User

The log shows ChatGPT-User opening the "small business tax prep" page about 15 times a month. She makes sure that page states prices, service area and how to book, since assistants are reading it to answer real questions.

FAQ

Does Google Analytics show AI bots?

No. Google Analytics automatically excludes known bots, and you can't turn that off. Use server logs or a firewall dashboard instead.

How often should I check my logs?

After any change to your host, firewall or robots.txt, and otherwise every few months. Turn on log archiving so you have history to compare.

Why does Google-Extended never show up in my logs?

Because it isn't a crawler. It's a robots.txt setting about how Google may use pages Googlebot already crawled, so the visits appear as Googlebot.

Can a bot ignore my robots.txt?

User-triggered fetchers may. OpenAI says robots.txt rules "may not apply" to ChatGPT-User; Perplexity says Perplexity-User "generally ignores robots.txt rules." Fake bots ignore everything.

Different for you?

Keep learning

NextCrawl efficiency: stop Google wasting crawls on junk URLsCrawl efficiency decides how fast Google indexes your pages, not your page views. Which filter, sort and duplicate URLs waste crawls, and how to stop them.

Do it yourself

Crawlability checklist: make every page reachable by GoogleA crawlability checklist for a small business site: find orphan pages, robots.txt blocks, broken links, redirect chains and server errors, with free tools.

Ask us one question

One line is enough. We reply by email.

Sources

  1. OpenAI: Overview of OpenAI Crawlers
  2. Claude Help Center: Does Anthropic crawl data from the web, and how can site owners block the crawler?
  3. Anthropic: crawler IP list (bots.json)
  4. Perplexity Docs: Perplexity crawlers
  5. Google Crawling Infrastructure: Verify requests from Google crawlers and fetchers
  6. Google Crawling Infrastructure: Google's common crawlers
  7. Google Crawling Infrastructure: Optimize your crawl budget
  8. Google Crawling Infrastructure: Myths about crawling
  9. Apple Support: About Applebot
  10. Bing: Bingbot IP list (bingbot.json)
  11. Bing Webmaster Guidelines
  12. Google Search Central: AI features and your website
  13. Analytics Help: Known bot-traffic exclusion
  14. cPanel Docs: Raw Access
  15. Apache HTTP Server Docs: Log Files
  16. Cloudflare Docs: AI Crawl Control