Crawl efficiency: stop Google wasting crawls on junk URLs
Short answer Crawl efficiency is how much of Googlebot's time goes to pages you want in search instead of duplicate or junk URLs. It affects how fast pages get indexed, not how many people view them: Google says crawling isn't a ranking signal. Under about 1,000 pages, Google says you needn't worry. Larger sites with filters should block the junk URLs.
Want help applying this to your business? Talk it through on a free 30-minute call.
Book a callDoes crawl efficiency increase page views?
Not directly. Crawling is Googlebot, Google's robot, fetching your pages. A page must be crawled before it can be indexed (stored and eligible to show in results), but being crawled more doesn't move it up. Google's myths page is direct:
"Improving your crawl rate won't necessarily lead to better positions in Google Search results. Google uses many signals to rank the results, and while crawling is necessary for a page to be in search results, it's not a ranking signal." (Google Crawling Infrastructure: Myths and facts about crawling)
What better crawling can do: get new and changed pages into Google sooner, and reach pages Google wasn't getting to at all. Views come after that, from ranking and from searchers choosing your result.
Does my site need to worry about crawl budget?
Crawl budget is the set of URLs Google can and wants to crawl on your site. For most small business sites, it's not a problem. Google's Crawl Stats help page says:
"If you have a site with fewer than a thousand pages, you should not need to use this report or worry about this level of crawling detail." (Search Console Help: Crawl Stats report)
Google's crawl budget guide is written for three kinds of sites. Google calls the numbers rough estimates, not exact thresholds.
| Your site | Should you work on crawl efficiency? |
|---|---|
| Under about 1,000 pages, new pages crawled within days | No. Keep your sitemap current and fix errors you see |
| 1 million+ pages that change about weekly | Yes, Google's guide is for you |
| 10,000+ pages that change daily (listings, inventory, events) | Yes |
| A large share of URLs marked "Discovered - currently not indexed" in Search Console | Yes, check for junk URLs first |
Google also says that if your pages "seem to be crawled the same day that they are published, you don't need to read this guide" (Google Crawling Infrastructure: Optimize your crawl budget). A small site can still create thousands of junk URLs, though. A store with color, size and price filters, or an events calendar, can turn 50 real pages into 50,000 addresses.
Which URLs waste crawls?
URLs that show the same or nearly the same content as another page. Google says the factor you control most is your "perceived inventory": without guidance, it tries to crawl most URLs it knows about, and duplicates waste "a lot of Google crawling time" (Google Crawling Infrastructure: Optimize your crawl budget).
| URL type | What it looks like | Fix |
|---|---|---|
| Filters (faceted navigation, the "narrow by" menus) | `/shop?color=red&size=m` | Block in robots.txt, or keep filter URLs tidy if you want them indexed |
| Sort orders | `/shop?sort=price` | Block in robots.txt |
| Ad and tracking tags | `/page?gclid=ABCD` | Canonical tag pointing to the clean page |
| Session IDs | `/page?sessionid=6EE2` | Store sessions in cookies, not the URL |
| Your site's own search results | `/?s=roof+repair` | Block in robots.txt |
| Endless calendars | `/events?month=8&year=2041` | nofollow on future-date links, or block |
| Removed pages that return "200 OK" (soft 404) | A "not found" message on a live page | Return a real 404 or 410 code |
| Redirect chains | Page A to B to C | Redirect A straight to C |
Examples and fixes come from Google's URL structure, faceted navigation, canonical and crawl budget pages. Google calls filter URLs a source of "infinite URL spaces" that cause overcrawling and slower discovery of new pages (Google Crawling Infrastructure: Improve crawling of faceted navigation URLs).
How do I stop Google crawling junk URLs?
Work in this order. Each step comes from Google's crawl budget guide unless noted.
- Merge true duplicates. Google ranks its signals by strength: a redirect and a `rel="canonical"` tag (a line in the page code naming the main version) are strong; being listed in your sitemap is weak. Using two together raises the chance Google picks your version (Google Search Central: How to specify a canonical URL).
- Block URLs you never want crawled in robots.txt, the text file at `yourdomain.com/robots.txt` that tells crawlers where not to go. Google's own example blocks filter parameters with lines like `disallow: /*?*color=` while still allowing one unfiltered "show all" page.
- Return 404 or 410 for pages that are gone. Google calls a 404 "a strong signal not to crawl that URL again". For a filter combination with no results, Google says to serve a 404 on that URL rather than redirect to a general error page.
- Fix soft 404s and long redirect chains. Both keep getting crawled.
- Keep your sitemap current, with a last-modified (lastmod) date on pages that change.
- Keep the server fast and healthy. Google raises its crawl limit when responses stay quick and lowers it on slowdowns, server errors (5xx codes) or "too many requests" (429).
Google gives two ways to get more crawl budget overall: more server capacity if Google is hitting it, and content Google finds popular, useful and unique (Google Crawling Infrastructure: Optimize your crawl budget).
Robots.txt, noindex or canonical: which one?
They do different jobs.
| Goal | Use | Why |
|---|---|---|
| Stop Google spending crawls on a URL | robots.txt | Google doesn't fetch it at all |
| Keep a page out of search results | noindex tag | robots.txt "is not a mechanism for keeping a web page out of Google" (Introduction to robots.txt) |
| Point copies to one main page | Redirect or canonical tag | Merges links and signals into one URL |
Two traps:
- noindex doesn't save crawls. Google still requests the page, then drops it when it sees the tag, "wasting crawling time" (Google Crawling Infrastructure: Optimize your crawl budget).
- Don't block a page you've marked noindex.
"For the noindex rule to be effective, the page or resource must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawler." (Google Search Central: Block Search indexing with noindex)
A blocked URL can still show in results, address only, if other sites link to it (Introduction to robots.txt).
What doesn't waste crawl budget
- 404 and other 4xx errors don't, except 429. Google gets a code and no content (Myths and facts about crawling).
- crawl-delay in robots.txt does nothing on Google: its crawlers don't process that rule (same page).
- nofollow on links helps only if every link to that URL, on your site and the web, carries it (same page).
- Search Console's URL Parameters tool no longer exists. Google announced it would be removed a month after March 28, 2022, saying its crawlers "will learn how to deal with URL parameters automatically" (Google Search Central Blog).
- Using robots.txt to shift budget to other pages. Google won't move the freed budget unless it's already at your server's limit (Optimize your crawl budget).
How do I check how Google crawls my site?
Open the Crawl Stats report: in Search Console, Settings, then Crawl stats. It shows how many requests Google made and when, what your server answered, and any availability problems. It works only on a Domain property or a root-level URL property, and it counts each step of a redirect chain as a separate request (Search Console Help: Crawl Stats report).
Then check the Page indexing report for "Discovered - currently not indexed". What each status means is in crawled or discovered, currently not indexed. A tour of the other reports is in Google Search Console for a small business.
Example
Illustrative cases with invented numbers.
A roofer with 35 pages
New pages appear in Google within a few days. Nothing to do.
A home décor store with 400 products (Canadian dollars)
Filters for color, size, material and price (C$25 bands) produce over 60,000 URLs, and new products take weeks to appear. The owner blocks the filter and sort parameters in robots.txt, keeps one "all products" page per category, and lists only product and category pages in the sitemap.
A wedding venue with an events calendar (US dollars)
The calendar links to every future month without end, and Crawl Stats shows most requests going to empty months. The developer adds nofollow to future-month links and returns a 404 for months with no events. The $1,200 package pages get crawled again within days.
FAQ
Is crawl rate a Google ranking factor?
No. Google says crawling is needed to appear in results but "it's not a ranking signal" (Myths and facts about crawling).
Does page speed affect crawling?
Yes. Faster loading and rendering let Google crawl more; many server errors slow it down. Speed for visitors matters more, Google adds (Myths and facts about crawling). See Core Web Vitals.
Do subdomains share one crawl budget?
No. Google treats each hostname as its own site, so `www.example.com` and `shop.example.com` have separate budgets (Optimize your crawl budget).
Should I block AI crawlers to save crawl budget?
They aren't part of Google's budget. But Google crawls less when your server slows, so heavy bot traffic can still hurt. See blocking AI crawlers with robots.txt.
Different for you?
- Search Console lists pages as not indexed: crawled or discovered, currently not indexed.
- You run an online store with filters: ecommerce SEO.
- Traffic fell after a new site launched: traffic dropped after a website redesign.
- Your platform controls robots.txt for you: Wix, Squarespace or WordPress for SEO.
- You'd rather have the technical checks done for you: SEO & GEO Agent.