AI crawler logs: see which AI bots visit your site
Short answer Download your raw access log from your host (or open Cloudflare's AI Crawl Control) and search for the bot names: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot, Applebot. Check the IP against each company's published list, since names can be faked. Google Analytics won't show them: it drops known bots.
Want help applying this to your business? Talk it through on a free 30-minute call.
Book a callWhere do I find my server logs?
With your web host, not in Google Analytics. A server log (also called an access log) is a text file where your server writes one line for every request, from people and bots alike. Analytics tools only count visitors who run their tracking code, and Google Analytics goes further:
"In Google Analytics properties, traffic from known bots and spiders is automatically excluded." (Analytics Help: Known bot-traffic exclusion)
Google adds that you "cannot disable" this or see how much was removed. So the log is the only place AI bots show up, unless a firewall in front of your site counts them for you.
| Your setup | Where to look |
|---|---|
| Shared hosting with cPanel | Metrics, then Raw Access. It downloads a compressed .gz file of your access log, for Apache or NGINX servers (cPanel Docs: Raw Access) |
| Your own server (VPS) | The web server's access log file. Ask whoever set it up for the path |
| Site behind Cloudflare | AI Crawl Control. Cloudflare says it shows "which AI services are accessing your content" and is available on all plans (Cloudflare Docs: AI Crawl Control) |
| Wix, Squarespace or another hosted builder | Ask the platform's support whether raw logs are available |
Two cPanel details matter. Your host can switch the Raw Access screen off. And cPanel saves older logs only if you tick the archive box in that screen; another box deletes last month's archives at month end (cPanel Docs: Raw Access). Turn archiving on now if you want a few months to compare.
Which bot names to search for
Open the log in a text editor or spreadsheet and search for these names. Each company documents its bots, and what a visit means differs by bot.
| Search for | Company | What a visit means |
|---|---|---|
| GPTBot | OpenAI | Collecting pages that may train its models |
| OAI-SearchBot | OpenAI | Indexing pages for ChatGPT search answers |
| ChatGPT-User | OpenAI | A ChatGPT user's question just led it to your page |
| OAI-AdsBot | OpenAI | Checking a page you submitted as a ChatGPT ad |
| ClaudeBot | Anthropic | Collecting training data |
| Claude-SearchBot | Anthropic | Indexing pages for Claude's search answers |
| Claude-User | Anthropic | A Claude user's question led it to your page |
| PerplexityBot | Perplexity | Indexing pages for Perplexity answers |
| Perplexity-User | Perplexity | A Perplexity user's question led it to your page |
| Googlebot | Google Search; AI Overviews and AI Mode link only to pages in that index | |
| Applebot | Apple | Siri, Spotlight and Safari search |
| bingbot | Microsoft | Bing, whose index also feeds Copilot answers |
Sources: OpenAI: Overview of OpenAI Crawlers; Claude Help Center; Perplexity Docs; Google Search Central: AI features and your website; Apple Support: About Applebot; Bing Webmaster Guidelines.
Don't look for Google-Extended or Applebot-Extended. They are settings you write in robots.txt, not visitors. Google says Google-Extended "doesn't have a separate HTTP request user agent string" (Google's common crawlers), and Apple says "Applebot-Extended does not crawl webpages" (Apple Support: About Applebot). Their traffic appears as Googlebot and Applebot.
The "-User" lines are the most telling for a business. OpenAI says ChatGPT-User visits a page "when users ask ChatGPT or a CustomGPT a question." A hit on your pricing page from one of these means someone asked an assistant something and it opened your page to answer. Whether to block any of these bots is a separate question: should you block AI crawlers?
How to read one log line
Most hosts use what Apache calls the Combined Log Format (Apache HTTP Server Docs: Log Files). Each line holds the same fields in the same order:
| Field | Example | What it tells you |
|---|---|---|
| IP address | 203.0.113.7 | Who asked; use it to check the bot is real |
| Date and time | [16/Nov/2026:09:14:02 -0500] | When |
| Request | "GET /furnace-repair/ HTTP/1.1" | Which page |
| Status code | 200 | 200 is fine; 301 redirected; 403 blocked; 404 not found; 5xx server error |
| Size | 18342 | Bytes sent back |
| Referrer | "-" | The page that linked here; bots usually send none |
| User agent | "...compatible; GPTBot/1.4; +https://openai.com/gptbot" | The name the visitor gives itself |
The bot name sits inside the last field, the user agent. OpenAI notes the version number in it may change, so search for "GPTBot", not "GPTBot/1.4".
Is it really GPTBot? Check the IP address
Anyone can type "GPTBot" into a request, so a name alone proves nothing. Google's verification page exists for exactly this, in case "spammers or other troublemakers" are posing as Google (Verify requests from Google crawlers and fetchers). Check the line's IP address against the list the company publishes:
| Bot | How to check |
|---|---|
| OpenAI bots | IP lists at openai.com/gptbot.json, searchbot.json and chatgpt-user.json (OpenAI) |
| Anthropic bots | IP list at claude.com/crawling/bots.json. Anthropic: an IP on that list "indicates that the crawler is coming from Anthropic" |
| Perplexity bots | IP lists at perplexity.ai/perplexitybot.json and perplexity-user.json (Perplexity Docs) |
| Googlebot | Reverse DNS lookup must end in googlebot.com, google.com or googleusercontent.com, and the forward lookup must return the same IP (Google) |
| Applebot | Reverse DNS in applebot.apple.com, or the IP list on Apple's page (Apple Support) |
| Bingbot | IP list at bing.com/toolbox/bingbot.json |
Perplexity's advice applies to every list: "Always use the most current IP ranges from the official JSON endpoints." The ranges change. For a handful of lines, paste the IP into a CIDR checker (a free tool that says whether an address falls inside a range) next to the list. Google says the manual method "is sufficient for most use cases."
What do AI bot visits tell you?
Mostly whether bots can reach you, not how well you rank. Google is blunt about its own crawler:
"Google uses many signals to rank the results, and while crawling is necessary for a page to be in search results, it's not a ranking signal." (Google: Myths about crawling)
So a page hit 40 times isn't "preferred" over one hit twice. No AI company says crawl counts reflect how much it values a page. Read the log for these problems instead:
| What you see | What it likely means | What to do |
|---|---|---|
| No OAI-SearchBot, Claude-SearchBot or PerplexityBot lines at all | Something may block them, often a firewall or host bot setting | Check your firewall's AI bot settings and robots.txt |
| Search bots get 403 | Your server or firewall refuses them | Allow search bots; OpenAI recommends "allowing requests from our published IP ranges" |
| Bots hit 404 on old URLs | Links point to pages that moved | Redirect moved pages people still link to |
| 5xx errors | Your server failed | Ask your host; Google advises keeping server errors low |
| Main service pages never visited | Bots may not find them | Link them from the homepage and menu; check pages not indexed |
| One bot floods the server | Too many requests at once | ClaudeBot honors a Crawl-delay line in robots.txt (Claude Help Center) |
Do I need to worry about crawl budget?
Almost certainly not. Crawl budget is how many pages Google will crawl on a site in a given time. Google's guide is aimed at sites with "1 million+ unique pages" changing weekly, or 10,000+ changing daily, and says:
"If your site doesn't have a large number of pages that change rapidly, or if your pages seem to be crawled the same day that they are published, you don't need to read this guide." (Google: Optimize your crawl budget)
A 30-page service business site is far below that. For Googlebot, Google points owners to the Crawl Stats report in Search Console instead (Google: Myths about crawling); the Search Console guide covers the reports worth reading.
Can I upload my log to ChatGPT to analyze it?
You can, but strip the IP address column first. Every human visitor's IP is in that file, and you don't need it once the bots are verified. A spreadsheet works just as well for a small site: split the file into columns on spaces, filter the user agent column for each bot name, then count rows by page and status code.
Example
Illustrative cases. Numbers are invented for the example.
A roofer with no AI search bot visits
A month of logs shows Googlebot and GPTBot but no OAI-SearchBot or PerplexityBot. The host's "block AI scrapers" switch was on. After allowing search bots, OAI-SearchBot lines appear within a week, with status 200.
A dental clinic checking "GPTBot" traffic
Thousands of GPTBot lines a day slow the site. Half the IPs aren't on OpenAI's list, so they're fakes posing as GPTBot. The clinic blocks those IPs at the firewall and leaves the real ones alone.
An accountant who sees ChatGPT-User
The log shows ChatGPT-User opening the "small business tax prep" page about 15 times a month. She makes sure that page states prices, service area and how to book, since assistants are reading it to answer real questions.
FAQ
Does Google Analytics show AI bots?
No. Google Analytics automatically excludes known bots, and you can't turn that off. Use server logs or a firewall dashboard instead.
How often should I check my logs?
After any change to your host, firewall or robots.txt, and otherwise every few months. Turn on log archiving so you have history to compare.
Why does Google-Extended never show up in my logs?
Because it isn't a crawler. It's a robots.txt setting about how Google may use pages Googlebot already crawled, so the visits appear as Googlebot.
Can a bot ignore my robots.txt?
User-triggered fetchers may. OpenAI says robots.txt rules "may not apply" to ChatGPT-User; Perplexity says Perplexity-User "generally ignores robots.txt rules." Fake bots ignore everything.
Different for you?
- You want to block some of these bots: should you block AI crawlers?
- Bots visit but ChatGPT still doesn't mention you: where ChatGPT finds local businesses.
- Google crawls your pages but doesn't index them: crawled or discovered, currently not indexed.
- You'd rather have the logs read and fixes made for you: SEO & GEO Agent.