Products SEO & GEO Agent Paid Ads Agent Website Agent Conversion Agent
How it works Pricing FAQ Blog Book a call

Search engine crawlers: how bots find and read your site

Updated · 7 min read

Short answer A crawler is a program that follows links from page to page and downloads what it finds for the search engine's index. Help it with plain links to every page that matters and, on bigger sites, a sitemap. Control it with robots.txt (what it may fetch) and noindex (what may appear in results). Most small sites need nothing more.

Want help applying this to your business? Talk it through on a free 30-minute call.

Book a call

How does a search engine crawler work?

A crawler (also called a bot or spider) is software that downloads web pages so a search engine can decide whether to store them in its index, the database results come from. Google's version is Googlebot. Google describes three stages, and a page can drop out at any of them:

StageWhat happensWhat stops it
DiscoveryGoogle learns a URL exists, mostly from links on pages it already knows, or from a sitemapNo link points to the page
CrawlingGooglebot fetches the page and runs its JavaScript in a recent version of Chromerobots.txt block, server errors, login wall
IndexingGoogle reads the text, title and images, picks one version among duplicates, and may store itnoindex tag, low-quality content

Ranking comes after all three, and no one can buy their way through them:

"Google doesn't accept payment to crawl a site more frequently, or rank it higher. If anyone tells you otherwise, they're wrong." (Google Search Central: In-depth guide to how Google Search works)

Three facts about Googlebot shape what you build, per Google's Googlebot page:

  • It mostly browses as a phone. Google indexes the mobile version of most sites, so most visits come from Googlebot Smartphone. Content missing on your mobile layout may be missed.
  • It reads only the start of big files. It fetches the first 2MB of a page or other supported file (64MB for a PDF) and ignores the rest. Older articles often quote 15MB; the current page says 2MB.
  • It visits gently. On most sites, about once every few seconds on average.

Which crawlers visit a small business website?

Your hosting or analytics logs show them by name (the "user agent"). The ones that matter for search:

CrawlerWho runs itWhat it feeds
Googlebot Smartphone and DesktopGoogleGoogle Search, Discover, Images, News
BingbotMicrosoftBing
ApplebotAppleSpotlight, Siri and Safari search
AI bots (GPTBot, ClaudeBot, PerplexityBot and others)AI companiesAI training or AI answers

Apple says Applebot's data powers search in "Spotlight, Siri, and Safari". Whether to let AI bots in is its own decision: see should you block AI crawlers?.

How do crawlers find new pages?

Mostly by following links. Google is plain about it:

"Every page you care about should have a link from at least one other page on your site." (Google Search Central: Make your links crawlable)

The link has to be a real one. Google says it can generally crawl a link only when it's a standard HTML link tag (an "a" element) with an href address. A button that jumps to a page only through a script may never be followed. Most website builders make proper links; custom-coded menus are where this breaks.

Other ways in, from most to least useful for a small site:

  1. A sitemap, a file listing your page addresses. Google says a site of about 500 pages or fewer that links its pages well may not need one, and that WordPress, Wix and similar systems usually publish one for you. Google ignores the `priority` and `changefreq` values in it.
  2. Request indexing in Search Console's URL Inspection tool, for one new or changed page. Google warns crawling "can take anywhere from a few days to a few weeks", there is a quota, and asking twice doesn't speed it up.
  3. IndexNow, a ping that tells Bing, Yandex, Naver, Seznam, Yep and others a page changed. One submission is shared with all of them. Google is not on its list of participating engines.

New page not showing after a few weeks? That's an indexing question: see Crawled or discovered, currently not indexed.

How do I control what crawlers do?

Pick the tool by what you want, because they do different jobs:

You wantUseWhy
A crawler to skip some URLs (search results pages, a calendar with endless dates)robots.txt, a text file at yoursite.com/robots.txtSaves crawling; doesn't hide the page
A page kept out of search results (thank-you page, staff login page)noindex tag in the page's codeGoogle drops it the next time it crawls the page
A page no one outside can open (client files)Password protectionThe only one that keeps it private

The mistake to avoid is using robots.txt to hide pages:

"it is not a mechanism for keeping a web page out of Google. To keep a web page out of Google, block indexing with noindex or password-protect the page." (Google Search Central: Introduction to robots.txt)

Don't combine the two either. Google says a page blocked by robots.txt is never crawled, so the crawler never sees its noindex, and the page can still show up if other sites link to it.

One more robots.txt trap: if the file returns a server error, Google's robots.txt rules say it stops crawling the whole site for the first 12 hours, then uses the last good copy for up to 30 days. A missing file (a 404) is fine; Google then assumes no limits.

Does a small business need to worry about crawl budget?

Almost never. Crawl budget is how many URLs Google can and wants to crawl on your site. Google's own guide says who it's for: sites with 1 million or more pages that change weekly, sites with 10,000 or more pages that change daily, or sites where much of the content sits in "Discovered - currently not indexed". It adds:

"If your site doesn't have a large number of pages that change rapidly, or if your pages seem to be crawled the same day that they are published, you don't need to read this guide." (Google Search Central: Crawl budget management for large sites)

What still helps any site: a fast, stable server (Google raises its crawl limit when responses stay quick) and a real 404 or 410 code for pages you deleted.

What if crawlers are slowing my site down?

First check the visitor is real. Google says its user-agent name "is often spoofed", so a bot calling itself Googlebot may not be one. A real one passes a reverse DNS lookup: the IP address in your log resolves to a name ending in googlebot.com, google.com or googleusercontent.com, and that name resolves back to the same IP. Apple's real crawler resolves to applebot.apple.com. Your host can run this check.

If it really is Googlebot and your server can't cope:

  • The Search Console crawl rate setting is gone. Google removed the crawl rate limiter tool on January 8, 2024.
  • Crawl-delay in robots.txt does nothing for Google. Its robots.txt rules list crawl-delay among fields that "aren't supported".
  • For an emergency, return error code 500, 503 or 429 for a few hours or 1–2 days. Google slows down and speeds up again once the errors stop. Longer than that, and pages may drop out of Google's index.
  • Look for the cause. Google says sudden crawl spikes usually come from filter and sort options or calendars that create endless URLs. Block those in robots.txt.

Example

Illustrative cases with invented numbers.

A plumber's 25-page WordPress site

The owner worries Google can't find his new water heater page. It's linked from the Services menu, and WordPress already publishes a sitemap. He requests indexing once in Search Console and waits. No crawl budget work, no sitemap tool, no paid "submission service".

A dental clinic's online booking calendar

The host reports heavy bot traffic. The logs show Googlebot fetching thousands of calendar date URLs. One robots.txt line blocking the calendar path ends it. The treatment pages stay crawlable.

A bookkeeper's client upload folder

A PDF from the client folder shows up in Google. Robots.txt won't fix that. She puts the folder behind a password, then asks Google to drop the old URL; see remove information from Google.

FAQ

How often does Googlebot crawl my site?

Google doesn't publish a schedule. Googlebot visits most sites no more than once every few seconds on average; more popular pages get crawled more often.

Can I pay Google to crawl my site faster?

No. Google says it doesn't accept payment to crawl more often or rank higher. Anyone selling "Google submission" is selling something free.

Does Googlebot see content loaded by JavaScript?

Usually. Google renders pages in a recent Chrome, sometimes after a delay. Google calls server-side rendering "a great idea": not every bot runs JavaScript.

Should I block bad bots in robots.txt?

Not much. Google says robots.txt "cannot enforce crawler behavior"; respectable crawlers obey it. Scrapers that ignore it need a firewall or your host's bot protection.

Different for you?

Keep learning

NextSearch engine saturation: what it means and how to measure itSearch engine saturation is the share of your pages Google has indexed. How to measure it in Search Console, what a healthy number looks like, and its limits.

Do it yourself

Crawlability checklist: make every page reachable by GoogleA crawlability checklist for a small business site: find orphan pages, robots.txt blocks, broken links, redirect chains and server errors, with free tools.

Ask us one question

One line is enough. We reply by email.

Sources

  1. Google Search Central: In-depth guide to how Google Search works
  2. Google Search Central: Googlebot
  3. Google Search Central: Make your links crawlable
  4. Google Search Central: Understand JavaScript SEO basics
  5. Google Search Central: What is a sitemap
  6. Google Search Central: Build and submit a sitemap
  7. Google Search Central: Ask Google to recrawl your URLs
  8. Google Search Central: Introduction to robots.txt
  9. Google Search Central: How Google interprets the robots.txt specification
  10. Google Search Central: Block Search indexing with noindex
  11. Google Search Central: Crawl budget management for large sites
  12. Google Crawling Infrastructure: Reduce the Google crawl rate
  13. Google Search Central Blog: Upcoming deprecation of Crawl Rate Limiter Tool in Search Console (November 2023)
  14. Google Crawling Infrastructure: Verify requests from Google crawlers and fetchers
  15. Apple Support: About Applebot
  16. Bing Webmaster Blog: New options for webmasters to control usage of their content in Bing Chat (2023)
  17. IndexNow: FAQ
  18. IndexNow: participating search engines list