Search engine crawlers: how bots find and read your site
Short answer A crawler is a program that follows links from page to page and downloads what it finds for the search engine's index. Help it with plain links to every page that matters and, on bigger sites, a sitemap. Control it with robots.txt (what it may fetch) and noindex (what may appear in results). Most small sites need nothing more.
Want help applying this to your business? Talk it through on a free 30-minute call.
Book a callHow does a search engine crawler work?
A crawler (also called a bot or spider) is software that downloads web pages so a search engine can decide whether to store them in its index, the database results come from. Google's version is Googlebot. Google describes three stages, and a page can drop out at any of them:
| Stage | What happens | What stops it |
|---|---|---|
| Discovery | Google learns a URL exists, mostly from links on pages it already knows, or from a sitemap | No link points to the page |
| Crawling | Googlebot fetches the page and runs its JavaScript in a recent version of Chrome | robots.txt block, server errors, login wall |
| Indexing | Google reads the text, title and images, picks one version among duplicates, and may store it | noindex tag, low-quality content |
Ranking comes after all three, and no one can buy their way through them:
"Google doesn't accept payment to crawl a site more frequently, or rank it higher. If anyone tells you otherwise, they're wrong." (Google Search Central: In-depth guide to how Google Search works)
Three facts about Googlebot shape what you build, per Google's Googlebot page:
- It mostly browses as a phone. Google indexes the mobile version of most sites, so most visits come from Googlebot Smartphone. Content missing on your mobile layout may be missed.
- It reads only the start of big files. It fetches the first 2MB of a page or other supported file (64MB for a PDF) and ignores the rest. Older articles often quote 15MB; the current page says 2MB.
- It visits gently. On most sites, about once every few seconds on average.
Which crawlers visit a small business website?
Your hosting or analytics logs show them by name (the "user agent"). The ones that matter for search:
| Crawler | Who runs it | What it feeds |
|---|---|---|
| Googlebot Smartphone and Desktop | Google Search, Discover, Images, News | |
| Bingbot | Microsoft | Bing |
| Applebot | Apple | Spotlight, Siri and Safari search |
| AI bots (GPTBot, ClaudeBot, PerplexityBot and others) | AI companies | AI training or AI answers |
Apple says Applebot's data powers search in "Spotlight, Siri, and Safari". Whether to let AI bots in is its own decision: see should you block AI crawlers?.
How do crawlers find new pages?
Mostly by following links. Google is plain about it:
"Every page you care about should have a link from at least one other page on your site." (Google Search Central: Make your links crawlable)
The link has to be a real one. Google says it can generally crawl a link only when it's a standard HTML link tag (an "a" element) with an href address. A button that jumps to a page only through a script may never be followed. Most website builders make proper links; custom-coded menus are where this breaks.
Other ways in, from most to least useful for a small site:
- A sitemap, a file listing your page addresses. Google says a site of about 500 pages or fewer that links its pages well may not need one, and that WordPress, Wix and similar systems usually publish one for you. Google ignores the `priority` and `changefreq` values in it.
- Request indexing in Search Console's URL Inspection tool, for one new or changed page. Google warns crawling "can take anywhere from a few days to a few weeks", there is a quota, and asking twice doesn't speed it up.
- IndexNow, a ping that tells Bing, Yandex, Naver, Seznam, Yep and others a page changed. One submission is shared with all of them. Google is not on its list of participating engines.
New page not showing after a few weeks? That's an indexing question: see Crawled or discovered, currently not indexed.
How do I control what crawlers do?
Pick the tool by what you want, because they do different jobs:
| You want | Use | Why |
|---|---|---|
| A crawler to skip some URLs (search results pages, a calendar with endless dates) | robots.txt, a text file at yoursite.com/robots.txt | Saves crawling; doesn't hide the page |
| A page kept out of search results (thank-you page, staff login page) | noindex tag in the page's code | Google drops it the next time it crawls the page |
| A page no one outside can open (client files) | Password protection | The only one that keeps it private |
The mistake to avoid is using robots.txt to hide pages:
"it is not a mechanism for keeping a web page out of Google. To keep a web page out of Google, block indexing with noindex or password-protect the page." (Google Search Central: Introduction to robots.txt)
Don't combine the two either. Google says a page blocked by robots.txt is never crawled, so the crawler never sees its noindex, and the page can still show up if other sites link to it.
One more robots.txt trap: if the file returns a server error, Google's robots.txt rules say it stops crawling the whole site for the first 12 hours, then uses the last good copy for up to 30 days. A missing file (a 404) is fine; Google then assumes no limits.
Does a small business need to worry about crawl budget?
Almost never. Crawl budget is how many URLs Google can and wants to crawl on your site. Google's own guide says who it's for: sites with 1 million or more pages that change weekly, sites with 10,000 or more pages that change daily, or sites where much of the content sits in "Discovered - currently not indexed". It adds:
"If your site doesn't have a large number of pages that change rapidly, or if your pages seem to be crawled the same day that they are published, you don't need to read this guide." (Google Search Central: Crawl budget management for large sites)
What still helps any site: a fast, stable server (Google raises its crawl limit when responses stay quick) and a real 404 or 410 code for pages you deleted.
What if crawlers are slowing my site down?
First check the visitor is real. Google says its user-agent name "is often spoofed", so a bot calling itself Googlebot may not be one. A real one passes a reverse DNS lookup: the IP address in your log resolves to a name ending in googlebot.com, google.com or googleusercontent.com, and that name resolves back to the same IP. Apple's real crawler resolves to applebot.apple.com. Your host can run this check.
If it really is Googlebot and your server can't cope:
- The Search Console crawl rate setting is gone. Google removed the crawl rate limiter tool on January 8, 2024.
- Crawl-delay in robots.txt does nothing for Google. Its robots.txt rules list crawl-delay among fields that "aren't supported".
- For an emergency, return error code 500, 503 or 429 for a few hours or 1–2 days. Google slows down and speeds up again once the errors stop. Longer than that, and pages may drop out of Google's index.
- Look for the cause. Google says sudden crawl spikes usually come from filter and sort options or calendars that create endless URLs. Block those in robots.txt.
Example
Illustrative cases with invented numbers.
A plumber's 25-page WordPress site
The owner worries Google can't find his new water heater page. It's linked from the Services menu, and WordPress already publishes a sitemap. He requests indexing once in Search Console and waits. No crawl budget work, no sitemap tool, no paid "submission service".
A dental clinic's online booking calendar
The host reports heavy bot traffic. The logs show Googlebot fetching thousands of calendar date URLs. One robots.txt line blocking the calendar path ends it. The treatment pages stay crawlable.
A bookkeeper's client upload folder
A PDF from the client folder shows up in Google. Robots.txt won't fix that. She puts the folder behind a password, then asks Google to drop the old URL; see remove information from Google.
FAQ
How often does Googlebot crawl my site?
Google doesn't publish a schedule. Googlebot visits most sites no more than once every few seconds on average; more popular pages get crawled more often.
Can I pay Google to crawl my site faster?
No. Google says it doesn't accept payment to crawl more often or rank higher. Anyone selling "Google submission" is selling something free.
Does Googlebot see content loaded by JavaScript?
Usually. Google renders pages in a recent Chrome, sometimes after a delay. Google calls server-side rendering "a great idea": not every bot runs JavaScript.
Should I block bad bots in robots.txt?
Not much. Google says robots.txt "cannot enforce crawler behavior"; respectable crawlers obey it. Scrapers that ignore it need a firewall or your host's bot protection.
Different for you?
- Your site isn't on Google at all: website not showing up on Google.
- Search Console lists pages as not indexed: Crawled or discovered, currently not indexed.
- You're deciding about GPTBot, ClaudeBot and other AI bots: should you block AI crawlers?.
- You just moved or rebuilt your site: traffic dropped after a redesign.
- You want to read Google's reports yourself: Google Search Console for a small business.
- You'd rather have the technical checks done for you: SEO & GEO Agent.