Crawl budget: does a small website need to worry?
Short answer Almost never. Google says its crawl budget guide is for sites with about 1 million pages that change weekly, or 10,000 that change daily. A service business site with dozens or hundreds of pages gets crawled in full. If a page is missing from Google, look at quality, duplicates or a block first, not crawl budget.
Want help applying this to your business? Talk it through on a free 30-minute call.
Book a callDoes a small website need to worry about crawl budget?
No. Google's crawl budget guide opens by telling most sites to skip it:
"If your site doesn't have a large number of pages that change rapidly, or if your pages seem to be crawled the same day that they are published, you don't need to read this guide." (Google Crawling Infrastructure: Optimize your crawl budget)
For everyone else, Google says an up-to-date sitemap and a regular look at the Page indexing report in Search Console are "adequate". The guide names who it's written for, and calls the numbers rough estimates, not exact cutoffs. The middle row is our reading; Google names no number there:
| Your site | Does crawl budget matter? |
|---|---|
| Under 1,000 pages (most service businesses, clinics, firms) | No |
| 1,000 to 10,000 pages that change now and then | Rarely |
| 10,000+ pages that change daily (listings, classifieds, big stores) | Yes |
| 1 million+ pages that change weekly | Yes |
| Many URLs stuck in "Discovered - currently not indexed" | Check it |
Google's Search Console help says the same about its Crawl Stats report: a site "with fewer than a thousand pages" should not need it.
What is crawl budget?
It's how many pages Google can and wants to fetch from your site. Google can't crawl the whole web, so it rations its time per site. Two things set your share:
- Crawl capacity limit: how hard Google can push your server without slowing it down. Every site starts at the same cautious level. If your server answers fast and without errors, the limit rises. If it slows or returns errors, Google backs off.
- Crawl demand: how much Google wants your pages. Google says this depends on popularity, how often pages change, and how many URLs it knows about on your site.
Google counts each hostname as its own site. So `www.example.com` and `shop.example.com` each get their own budget.
Crawling is not indexing. Google crawls a page (downloads it), then decides whether to keep it in search results. A page can be crawled fine and still be left out. For that problem, see crawled or discovered, currently not indexed.
When does crawl budget matter?
When a site creates far more URLs than it has real pages. Google's crawl-rate guide names the usual culprits:
- Filters and sort options that make a new URL for every combination ("faceted navigation", common on online stores)
- Calendars with a page for every date
- Endless scroll or search-result pages that repeat content found elsewhere
A store with a few hundred products and several filters can expose far more URLs than it has products. Google may then spend its time on filter pages and reach new products late. Signs you have this problem:
- Search Console shows a large share of pages as "Discovered - currently not indexed".
- New pages take weeks to be crawled, not days.
- URL Inspection in Search Console reports "Hostload exceeded", which Google says means your server, not your content, is the limit.
Does crawl budget affect rankings?
No. Google answers this directly:
"Improving your crawl rate won't necessarily lead to better positions in Google Search results. Google uses many signals to rank the results, and while crawling is necessary for a page to be in search results, it's not a ranking signal." (Google Crawling Infrastructure: Myths and facts about crawling)
A page has to be crawled once to appear at all. After that, more crawling doesn't move it up. Size doesn't decide crawl frequency either: Google says pages with important content that changes often get crawled often, "regardless of the size" of the site.
What can slow Google down on a small site?
Server trouble, not page count. Google's own pages list the problems that make it crawl less:
| Problem | What Google does |
|---|---|
| Slow server responses | Crawls less |
| Many server errors (5xx) or "too many requests" (429) | Crawls the whole hostname less until errors drop |
| robots.txt can't be fetched (server error) | Stops crawling for the first 12 hours, then uses the last good copy for up to 30 days |
| Errors on the same URL for several days | May drop that URL from the index |
| Long redirect chains | Google says they hurt crawling |
A missing robots.txt is fine. Google treats "file not found" (404) as no rules, so it crawls everything. The danger is a robots.txt address that times out or errors, which some cheap hosting does during outages. If you're unsure, ask your host or web person to check that the file loads.
How do I check how often Google crawls my site?
Open Search Console, then Settings, then Crawl stats. It shows requests per day, average response time and whether Google hit errors. For a small site, the useful part is Host status: green means Google reached your robots.txt, DNS (the system that points your domain to your server) and server without trouble.
Two quicker checks answer most owner questions:
- Is one page crawled? Paste its address into URL Inspection. It shows the last crawl date.
- Are pages missing? Open Page indexing and read the reasons. See Google Search Console for a small business.
How to fix crawl budget on a large site
If you are in the "Yes" rows above, these are Google's recommendations, in plain words:
| Do | Why |
|---|---|
| Merge duplicate pages into one | Google crawls URLs, so fewer copies means more time on real pages |
| Block endless filter or sort URLs in robots.txt | Stops the fetches entirely |
| Return 404 or 410 for pages gone for good | Google says a 404 is "a strong signal not to crawl that URL again" |
| Fix "soft 404s" (empty pages that answer as if fine) | Google keeps crawling them |
| Keep the sitemap current, with last-changed dates | Points Google to new and updated pages |
| Speed up the server and support 304 "not modified" replies | Lets Google skip pages that didn't change |
Three traps. First, `noindex` doesn't save crawling: Google still fetches the page to read the tag. Use it to keep a page out of results, not to save budget. Second, blocking pages in robots.txt for a while won't push Google to other pages. Google says it won't shift that freed budget unless your server was already at its limit. Third, don't block pages you want in Google, or the scripts and styles Google needs to understand them.
You can't ask Google to crawl more. Google says the two ways to more budget are a stronger server and more useful, unique content.
Example
Illustrative cases with invented numbers.
A plumber with 40 pages
The site has a home page, 12 service pages, 20 area pages and a few posts. Search Console shows Google fetches about 50 URLs a day. Crawl budget is no issue. Three area pages show "Crawled - currently not indexed" because they repeat each other. That is a content problem, not a crawl problem.
An online store with filters (Canadian dollars)
A furniture store sells 2,000 products. Color, size and price filters create over 100,000 URLs. New sofas take a month to appear in Google. Blocking the filter URLs in robots.txt and keeping one page per category brings new products in within days. The developer bills about C$800 for the change.
A clinic whose site went down for a weekend
The host had an outage from Friday night to Sunday. Google slowed its crawling, then picked back up once the server answered normally. Rankings returned within the week. Nothing needed fixing beyond the host.
FAQ
Should my important pages be within three clicks of the home page?
It helps people and crawlers find them. Google says pages linked from the home page may be crawled more often, but that "doesn't mean that these pages will be ranked more highly".
Can I use crawl-delay in robots.txt to control Google?
No. Google says its crawlers don't process the "crawl-delay" rule.
Do 404 pages waste crawl budget?
No. Google says pages that return 4xx codes (except 429) don't waste crawl budget, because Google got a status code and no content.
Does updating my page date make Google crawl it more?
Google says trivial changes and new dates add nothing: content is judged by quality, regardless of age. Update a page when the facts change.
Do I need a sitemap for a small site?
Not necessarily. Google says a site of "about 500 pages or fewer" that links its pages well might not need one. It's still free and quick to add.
Different for you?
- Pages show "Discovered" or "Crawled - currently not indexed": crawled or discovered, currently not indexed.
- Your site isn't in Google at all: website not showing on Google.
- You run an online store with filters: ecommerce SEO.
- Your server or site is slow: Core Web Vitals and site speed.
- You want AI crawlers handled too: should you block AI crawlers?.
- You'd rather have the technical checks done for you: SEO & GEO Agent.