Products SEO & GEO Agent Paid Ads Agent Website Agent Conversion Agent
How it works Pricing FAQ Blog Book a call

Crawl budget: does a small website need to worry?

Updated · 7 min read

Short answer Almost never. Google says its crawl budget guide is for sites with about 1 million pages that change weekly, or 10,000 that change daily. A service business site with dozens or hundreds of pages gets crawled in full. If a page is missing from Google, look at quality, duplicates or a block first, not crawl budget.

Want help applying this to your business? Talk it through on a free 30-minute call.

Book a call

Does a small website need to worry about crawl budget?

No. Google's crawl budget guide opens by telling most sites to skip it:

"If your site doesn't have a large number of pages that change rapidly, or if your pages seem to be crawled the same day that they are published, you don't need to read this guide." (Google Crawling Infrastructure: Optimize your crawl budget)

For everyone else, Google says an up-to-date sitemap and a regular look at the Page indexing report in Search Console are "adequate". The guide names who it's written for, and calls the numbers rough estimates, not exact cutoffs. The middle row is our reading; Google names no number there:

Your siteDoes crawl budget matter?
Under 1,000 pages (most service businesses, clinics, firms)No
1,000 to 10,000 pages that change now and thenRarely
10,000+ pages that change daily (listings, classifieds, big stores)Yes
1 million+ pages that change weeklyYes
Many URLs stuck in "Discovered - currently not indexed"Check it

Google's Search Console help says the same about its Crawl Stats report: a site "with fewer than a thousand pages" should not need it.

What is crawl budget?

It's how many pages Google can and wants to fetch from your site. Google can't crawl the whole web, so it rations its time per site. Two things set your share:

  • Crawl capacity limit: how hard Google can push your server without slowing it down. Every site starts at the same cautious level. If your server answers fast and without errors, the limit rises. If it slows or returns errors, Google backs off.
  • Crawl demand: how much Google wants your pages. Google says this depends on popularity, how often pages change, and how many URLs it knows about on your site.

Google counts each hostname as its own site. So `www.example.com` and `shop.example.com` each get their own budget.

Crawling is not indexing. Google crawls a page (downloads it), then decides whether to keep it in search results. A page can be crawled fine and still be left out. For that problem, see crawled or discovered, currently not indexed.

When does crawl budget matter?

When a site creates far more URLs than it has real pages. Google's crawl-rate guide names the usual culprits:

  • Filters and sort options that make a new URL for every combination ("faceted navigation", common on online stores)
  • Calendars with a page for every date
  • Endless scroll or search-result pages that repeat content found elsewhere

A store with a few hundred products and several filters can expose far more URLs than it has products. Google may then spend its time on filter pages and reach new products late. Signs you have this problem:

  1. Search Console shows a large share of pages as "Discovered - currently not indexed".
  2. New pages take weeks to be crawled, not days.
  3. URL Inspection in Search Console reports "Hostload exceeded", which Google says means your server, not your content, is the limit.

Does crawl budget affect rankings?

No. Google answers this directly:

"Improving your crawl rate won't necessarily lead to better positions in Google Search results. Google uses many signals to rank the results, and while crawling is necessary for a page to be in search results, it's not a ranking signal." (Google Crawling Infrastructure: Myths and facts about crawling)

A page has to be crawled once to appear at all. After that, more crawling doesn't move it up. Size doesn't decide crawl frequency either: Google says pages with important content that changes often get crawled often, "regardless of the size" of the site.

What can slow Google down on a small site?

Server trouble, not page count. Google's own pages list the problems that make it crawl less:

ProblemWhat Google does
Slow server responsesCrawls less
Many server errors (5xx) or "too many requests" (429)Crawls the whole hostname less until errors drop
robots.txt can't be fetched (server error)Stops crawling for the first 12 hours, then uses the last good copy for up to 30 days
Errors on the same URL for several daysMay drop that URL from the index
Long redirect chainsGoogle says they hurt crawling

A missing robots.txt is fine. Google treats "file not found" (404) as no rules, so it crawls everything. The danger is a robots.txt address that times out or errors, which some cheap hosting does during outages. If you're unsure, ask your host or web person to check that the file loads.

How do I check how often Google crawls my site?

Open Search Console, then Settings, then Crawl stats. It shows requests per day, average response time and whether Google hit errors. For a small site, the useful part is Host status: green means Google reached your robots.txt, DNS (the system that points your domain to your server) and server without trouble.

Two quicker checks answer most owner questions:

How to fix crawl budget on a large site

If you are in the "Yes" rows above, these are Google's recommendations, in plain words:

DoWhy
Merge duplicate pages into oneGoogle crawls URLs, so fewer copies means more time on real pages
Block endless filter or sort URLs in robots.txtStops the fetches entirely
Return 404 or 410 for pages gone for goodGoogle says a 404 is "a strong signal not to crawl that URL again"
Fix "soft 404s" (empty pages that answer as if fine)Google keeps crawling them
Keep the sitemap current, with last-changed datesPoints Google to new and updated pages
Speed up the server and support 304 "not modified" repliesLets Google skip pages that didn't change

Three traps. First, `noindex` doesn't save crawling: Google still fetches the page to read the tag. Use it to keep a page out of results, not to save budget. Second, blocking pages in robots.txt for a while won't push Google to other pages. Google says it won't shift that freed budget unless your server was already at its limit. Third, don't block pages you want in Google, or the scripts and styles Google needs to understand them.

You can't ask Google to crawl more. Google says the two ways to more budget are a stronger server and more useful, unique content.

Example

Illustrative cases with invented numbers.

A plumber with 40 pages

The site has a home page, 12 service pages, 20 area pages and a few posts. Search Console shows Google fetches about 50 URLs a day. Crawl budget is no issue. Three area pages show "Crawled - currently not indexed" because they repeat each other. That is a content problem, not a crawl problem.

An online store with filters (Canadian dollars)

A furniture store sells 2,000 products. Color, size and price filters create over 100,000 URLs. New sofas take a month to appear in Google. Blocking the filter URLs in robots.txt and keeping one page per category brings new products in within days. The developer bills about C$800 for the change.

A clinic whose site went down for a weekend

The host had an outage from Friday night to Sunday. Google slowed its crawling, then picked back up once the server answered normally. Rankings returned within the week. Nothing needed fixing beyond the host.

FAQ

Should my important pages be within three clicks of the home page?

It helps people and crawlers find them. Google says pages linked from the home page may be crawled more often, but that "doesn't mean that these pages will be ranked more highly".

Can I use crawl-delay in robots.txt to control Google?

No. Google says its crawlers don't process the "crawl-delay" rule.

Do 404 pages waste crawl budget?

No. Google says pages that return 4xx codes (except 429) don't waste crawl budget, because Google got a status code and no content.

Does updating my page date make Google crawl it more?

Google says trivial changes and new dates add nothing: content is judged by quality, regardless of age. Update a page when the facts change.

Do I need a sitemap for a small site?

Not necessarily. Google says a site of "about 500 pages or fewer" that links its pages well might not need one. It's still free and quick to add.

Different for you?

Keep learning

NextCrawl efficiency: does it really grow organic traffic?Crawl efficiency rarely limits a small site. When Google says crawl budget matters, how to check yours in Search Console, and the few fixes worth doing.

Do it yourself

Crawlability checklist: make every page reachable by GoogleA crawlability checklist for a small business site: find orphan pages, robots.txt blocks, broken links, redirect chains and server errors, with free tools.

Ask us one question

One line is enough. We reply by email.

Sources

  1. Google Crawling Infrastructure: Optimize your crawl budget
  2. Google Crawling Infrastructure: Myths and facts about crawling
  3. Google Crawling Infrastructure: Reduce the Google crawl rate
  4. Search Console Help: Crawl Stats report
  5. Google Search Central: Learn about sitemaps
  6. Google Search Central: Introduction to robots.txt
  7. Google Crawling Infrastructure: How Google interprets the robots.txt specification