Crawl budget optimization for large sites: fix wasted crawls
Short answer Pull 30 days of server logs, keep only verified Googlebot requests, and group them by URL pattern and status code. Every request to a URL you don't want in Google is waste. Filters, sort options, redirect chains, soft 404s and server errors are the usual culprits. Block, merge or fix them, then recount next month.
Want help applying this to your business? Talk it through on a free 30-minute call.
Book a callDoes your site need crawl budget optimization?
Only if it is large or changes fast. Google's crawl budget guide is written for sites with about 1 million unique pages that change weekly, sites with 10,000+ pages that change daily, and sites where Search Console lists a large share of URLs as "Discovered - currently not indexed". Google calls those numbers rough estimates, not cutoffs.
"If your site doesn't have a large number of pages that change rapidly, or if your pages seem to be crawled the same day that they are published, you don't need to read this guide." (Google Crawling Infrastructure: Optimize your crawl budget)
Think listings sites, big online stores, directories, multi-location brands with thousands of pages, and publishers. A 40-page service site doesn't need any of this. If its pages are missing from Google, see crawled or discovered, currently not indexed.
Crawl budget is the set of URLs Google can and wants to fetch from one hostname. Google says the part you control most is your "perceived inventory": every URL Google knows about. If many are duplicates or junk, Google spends its visits there instead of on your new pages.
How do I find wasted crawls in my server logs?
Count where Googlebot actually goes, then compare that with where you want it to go. Five steps:
- Get the logs. Ask your host, CDN or developer for 30 days of raw access logs. Each line records one request: time, address, URL, status code and user agent (the name the visitor gives itself).
- Keep only real Googlebot. Anyone can call their bot "Googlebot". Google's verification guide says to run a reverse DNS lookup on the requesting IP address, check the name ends in googlebot.com, google.com or googleusercontent.com, then look that name up again and confirm it returns the same IP. For large logs, match IPs against the Googlebot IP ranges Google publishes as a JSON file.
- Group by URL pattern and status code. Don't read URL by URL. Strip each URL to its pattern (`/listings/?sort=`, `/events/2031-`, `/product/*`) and count requests per pattern and per code (200, 301, 404, 5xx).
- Compare with the pages you want in Google. Your sitemap is that list. Every request to a URL outside it, or to a redirect or error, goes in the waste pile. Waste share = waste requests ÷ all Googlebot requests.
- Check discovery. For pages published last month, find the first Googlebot request in the log. Days from publish to first crawl is the number crawl budget work should shrink.
Small logs fit in a spreadsheet. Millions of lines need a log analyzer tool or a developer with a script.
Which URL patterns waste the most crawls?
These are the patterns Google's own pages name, and what each one looks like in a log:
| What you see in the log | What it is | Fix |
|---|---|---|
| Thousands of URLs with `?color=`, `?size=`, `?price=` | Filters (faceted navigation) making a URL for every combination | Block filter parameters in robots.txt; keep item pages and one unfiltered list |
| Same page with `?sort=` or session IDs | Duplicate copies of one page | Merge into one URL; block in robots.txt if you can't |
| Dates years ahead, `/calendar/2031-04-12` | Calendar with a page per day | Block the date pattern in robots.txt |
| Filter pages with no results answering 200 | Soft 404s | Return a real 404 status code |
| 301 to 301 to 200 | Redirect chains, each hop a separate request | Link and redirect straight to the final URL |
| Bursts of 5xx or 429, or slow response times | Server can't keep up, so Google lowers its crawl limit for the whole hostname | Speed up responses or add server capacity |
| Same unchanged pages crawled daily, all 200 | No caching signal | Support 304 "not modified" replies |
| Pages with a noindex tag crawled often | Google must fetch them to see the tag | Block in robots.txt if they never need crawling |
On filters, Google's faceted navigation guide is blunt:
"Oftentimes there's no good reason to allow crawling of filtered items, as it consumes server resources for no or negligible benefit." (Google Crawling Infrastructure: Managing crawling of faceted navigation URLs)
Leave 404s alone. Google says pages returning 4xx codes, except 429, don't waste crawl budget, because Google gets a status code and no content. CSS, JavaScript and alternate URLs (hreflang language versions) do count, per the same myths page.
No log access? Use the Crawl Stats report
Search Console's Crawl Stats report (Settings, then Crawl stats) gives a summary from Google's side. It splits requests by response code, file type, purpose and Googlebot type, with sample URLs for each. Limits to know:
- It works only for root-level properties, such as a Domain property.
- Sample URLs are "not comprehensive", Google says, and some requests may not be counted, so totals can differ from your logs.
- Each redirect hop counts as its own request, which makes chains easy to spot.
- "Discovery" means a URL Google had never crawled; "Refresh" is a recrawl. After you add many new pages or a sitemap, Google says you should see a bump in discovery crawls.
- A spike from "AdsBot" usually means many new Dynamic Search Ads targets. Google says AdsBot crawls those URLs about every two weeks.
The report helps. It can't tell you which of your URL patterns soak up visits. Only logs do that.
Which crawl budget fixes don't work?
Several common moves do nothing for Google, according to its own pages:
| Move | Why it fails |
|---|---|
| Adding noindex to save crawls | Google still fetches the page to read the tag |
| Blocking pages in robots.txt for a while to push Google elsewhere | Google won't shift that budget unless your server is already at its limit |
| crawl-delay in robots.txt | Google's crawlers don't process it |
| Compressing sitemaps | Google still fetches them |
| Changing page dates with no real edits | No added value; Google rates content by quality, not age |
| nofollow on internal links | The URL still gets crawled if any other link points to it |
Crawling isn't a ranking factor either. Google says a page must be crawled to appear, but more crawling doesn't lift its position. The payoff is new and changed pages reaching Google sooner.
What if Googlebot is crawling too much?
First find the cause in your logs: Google's crawl-rate guide says sudden spikes usually come from filters, date calendars or Dynamic Search Ads targets. For an emergency lasting hours or a day or two, Google says to return 500, 503 or 429 status codes to crawl requests. Google then slows down across the hostname and speeds back up as the errors stop.
"We don't recommend that you do this for a long period of time (meaning, longer than 1-2 days) as it may have a negative effect on how your site appears in Google products." (Google Crawling Infrastructure: Reduce the Google crawl rate)
If you can't serve errors, Google offers a form to report an unusually high crawl rate. There is no way to ask for more crawling.
How do I know the fix worked?
Recount a month later with the same method. Three numbers should move:
- Waste share falls.
- Days from publish to first crawl for new pages falls.
- Server errors in Crawl Stats stay low, and host status stays green.
Total Googlebot requests may drop. That's fine if the requests left go to pages you want found.
Example
Illustrative cases with invented numbers.
A real estate listings site (US dollars)
The site has 60,000 active listings. Logs show 1.2 million Googlebot requests in 30 days; 700,000 hit `?sort=` and map-view URLs. New listings wait 9 days for a first crawl. A developer blocks both patterns in robots.txt for about $1,500. Next month, waste share falls from 58% to 15% and new listings are crawled within 2 days.
An online tire store (Canadian dollars)
A store with 8,000 products shows "Discovered - currently not indexed" on 40% of its URLs. Logs show filter pages for sizes it doesn't stock, all answering 200 with "no results". Returning 404 for empty combinations costs about C$900 in development. Those URLs drop out of the crawl, and product pages are crawled more often.
A franchise with 2,000 location pages
Every location page links to an events calendar, creating a page for every day through 2035. Googlebot spends a third of its requests there. One robots.txt line blocking future dates frees those visits for the location pages.
FAQ
How many days of logs do I need?
Thirty days is a practical minimum: it covers weekly patterns and shows the publish-to-first-crawl delay for a month of new pages. Busy sites can learn a lot from a week.
Is every "Googlebot" in my logs really Google?
No. Scrapers fake the name. Google's guide says to confirm with a reverse and forward DNS lookup, or by matching the IP to Google's published IP ranges.
Should I block CSS and JavaScript to save crawl budget?
No. Google needs those files to render the page before indexing it. Its Crawl Stats help warns not to block resources Google needs to understand the content.
Do subdomains share one crawl budget?
No. Google treats each hostname as its own site, so `www.example.com` and `shop.example.com` each get a separate budget.
Will better links increase my crawl budget?
Indirectly. Google says more popular URLs get crawled more often. The two ways it names to get more budget are more server capacity and higher-quality content.
Different for you?
- Pages stuck as "Discovered" or "Crawled - currently not indexed": crawled or discovered, currently not indexed.
- An online store with filters and categories: ecommerce SEO.
- New to Search Console's reports: Google Search Console for a small business.
- Crawling and traffic dropped after a site move or redesign: traffic dropped after a redesign.
- AI bots filling your logs: should you block AI crawlers?.
- You'd rather have the log audit and fixes done for you: SEO & GEO Agent.