Products SEO & GEO Agent Paid Ads Agent Website Agent Conversion Agent
How it works Pricing FAQ Blog Book a call

Crawl efficiency: stop Google wasting crawls on junk URLs

Updated · 7 min read

Short answer Crawl efficiency is how much of Googlebot's time goes to pages you want in search instead of duplicate or junk URLs. It affects how fast pages get indexed, not how many people view them: Google says crawling isn't a ranking signal. Under about 1,000 pages, Google says you needn't worry. Larger sites with filters should block the junk URLs.

Want help applying this to your business? Talk it through on a free 30-minute call.

Book a call

Does crawl efficiency increase page views?

Not directly. Crawling is Googlebot, Google's robot, fetching your pages. A page must be crawled before it can be indexed (stored and eligible to show in results), but being crawled more doesn't move it up. Google's myths page is direct:

"Improving your crawl rate won't necessarily lead to better positions in Google Search results. Google uses many signals to rank the results, and while crawling is necessary for a page to be in search results, it's not a ranking signal." (Google Crawling Infrastructure: Myths and facts about crawling)

What better crawling can do: get new and changed pages into Google sooner, and reach pages Google wasn't getting to at all. Views come after that, from ranking and from searchers choosing your result.

Does my site need to worry about crawl budget?

Crawl budget is the set of URLs Google can and wants to crawl on your site. For most small business sites, it's not a problem. Google's Crawl Stats help page says:

"If you have a site with fewer than a thousand pages, you should not need to use this report or worry about this level of crawling detail." (Search Console Help: Crawl Stats report)

Google's crawl budget guide is written for three kinds of sites. Google calls the numbers rough estimates, not exact thresholds.

Your siteShould you work on crawl efficiency?
Under about 1,000 pages, new pages crawled within daysNo. Keep your sitemap current and fix errors you see
1 million+ pages that change about weeklyYes, Google's guide is for you
10,000+ pages that change daily (listings, inventory, events)Yes
A large share of URLs marked "Discovered - currently not indexed" in Search ConsoleYes, check for junk URLs first

Google also says that if your pages "seem to be crawled the same day that they are published, you don't need to read this guide" (Google Crawling Infrastructure: Optimize your crawl budget). A small site can still create thousands of junk URLs, though. A store with color, size and price filters, or an events calendar, can turn 50 real pages into 50,000 addresses.

Which URLs waste crawls?

URLs that show the same or nearly the same content as another page. Google says the factor you control most is your "perceived inventory": without guidance, it tries to crawl most URLs it knows about, and duplicates waste "a lot of Google crawling time" (Google Crawling Infrastructure: Optimize your crawl budget).

URL typeWhat it looks likeFix
Filters (faceted navigation, the "narrow by" menus)`/shop?color=red&size=m`Block in robots.txt, or keep filter URLs tidy if you want them indexed
Sort orders`/shop?sort=price`Block in robots.txt
Ad and tracking tags`/page?gclid=ABCD`Canonical tag pointing to the clean page
Session IDs`/page?sessionid=6EE2`Store sessions in cookies, not the URL
Your site's own search results`/?s=roof+repair`Block in robots.txt
Endless calendars`/events?month=8&year=2041`nofollow on future-date links, or block
Removed pages that return "200 OK" (soft 404)A "not found" message on a live pageReturn a real 404 or 410 code
Redirect chainsPage A to B to CRedirect A straight to C

Examples and fixes come from Google's URL structure, faceted navigation, canonical and crawl budget pages. Google calls filter URLs a source of "infinite URL spaces" that cause overcrawling and slower discovery of new pages (Google Crawling Infrastructure: Improve crawling of faceted navigation URLs).

How do I stop Google crawling junk URLs?

Work in this order. Each step comes from Google's crawl budget guide unless noted.

  1. Merge true duplicates. Google ranks its signals by strength: a redirect and a `rel="canonical"` tag (a line in the page code naming the main version) are strong; being listed in your sitemap is weak. Using two together raises the chance Google picks your version (Google Search Central: How to specify a canonical URL).
  2. Block URLs you never want crawled in robots.txt, the text file at `yourdomain.com/robots.txt` that tells crawlers where not to go. Google's own example blocks filter parameters with lines like `disallow: /*?*color=` while still allowing one unfiltered "show all" page.
  3. Return 404 or 410 for pages that are gone. Google calls a 404 "a strong signal not to crawl that URL again". For a filter combination with no results, Google says to serve a 404 on that URL rather than redirect to a general error page.
  4. Fix soft 404s and long redirect chains. Both keep getting crawled.
  5. Keep your sitemap current, with a last-modified (lastmod) date on pages that change.
  6. Keep the server fast and healthy. Google raises its crawl limit when responses stay quick and lowers it on slowdowns, server errors (5xx codes) or "too many requests" (429).

Google gives two ways to get more crawl budget overall: more server capacity if Google is hitting it, and content Google finds popular, useful and unique (Google Crawling Infrastructure: Optimize your crawl budget).

Robots.txt, noindex or canonical: which one?

They do different jobs.

GoalUseWhy
Stop Google spending crawls on a URLrobots.txtGoogle doesn't fetch it at all
Keep a page out of search resultsnoindex tagrobots.txt "is not a mechanism for keeping a web page out of Google" (Introduction to robots.txt)
Point copies to one main pageRedirect or canonical tagMerges links and signals into one URL

Two traps:

  • noindex doesn't save crawls. Google still requests the page, then drops it when it sees the tag, "wasting crawling time" (Google Crawling Infrastructure: Optimize your crawl budget).
  • Don't block a page you've marked noindex.
"For the noindex rule to be effective, the page or resource must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawler." (Google Search Central: Block Search indexing with noindex)

A blocked URL can still show in results, address only, if other sites link to it (Introduction to robots.txt).

What doesn't waste crawl budget

  • 404 and other 4xx errors don't, except 429. Google gets a code and no content (Myths and facts about crawling).
  • crawl-delay in robots.txt does nothing on Google: its crawlers don't process that rule (same page).
  • nofollow on links helps only if every link to that URL, on your site and the web, carries it (same page).
  • Search Console's URL Parameters tool no longer exists. Google announced it would be removed a month after March 28, 2022, saying its crawlers "will learn how to deal with URL parameters automatically" (Google Search Central Blog).
  • Using robots.txt to shift budget to other pages. Google won't move the freed budget unless it's already at your server's limit (Optimize your crawl budget).

How do I check how Google crawls my site?

Open the Crawl Stats report: in Search Console, Settings, then Crawl stats. It shows how many requests Google made and when, what your server answered, and any availability problems. It works only on a Domain property or a root-level URL property, and it counts each step of a redirect chain as a separate request (Search Console Help: Crawl Stats report).

Then check the Page indexing report for "Discovered - currently not indexed". What each status means is in crawled or discovered, currently not indexed. A tour of the other reports is in Google Search Console for a small business.

Example

Illustrative cases with invented numbers.

A roofer with 35 pages

New pages appear in Google within a few days. Nothing to do.

A home décor store with 400 products (Canadian dollars)

Filters for color, size, material and price (C$25 bands) produce over 60,000 URLs, and new products take weeks to appear. The owner blocks the filter and sort parameters in robots.txt, keeps one "all products" page per category, and lists only product and category pages in the sitemap.

A wedding venue with an events calendar (US dollars)

The calendar links to every future month without end, and Crawl Stats shows most requests going to empty months. The developer adds nofollow to future-month links and returns a 404 for months with no events. The $1,200 package pages get crawled again within days.

FAQ

Is crawl rate a Google ranking factor?

No. Google says crawling is needed to appear in results but "it's not a ranking signal" (Myths and facts about crawling).

Does page speed affect crawling?

Yes. Faster loading and rendering let Google crawl more; many server errors slow it down. Speed for visitors matters more, Google adds (Myths and facts about crawling). See Core Web Vitals.

Do subdomains share one crawl budget?

No. Google treats each hostname as its own site, so `www.example.com` and `shop.example.com` have separate budgets (Optimize your crawl budget).

Should I block AI crawlers to save crawl budget?

They aren't part of Google's budget. But Google crawls less when your server slows, so heavy bot traffic can still hurt. See blocking AI crawlers with robots.txt.

Different for you?

Keep learning

NextWebsite not showing up on Google: the checks in orderCan't find your website on Google? Run the checks in order: a site: search, URL Inspection, noindex and platform settings, new-site timing, Maps vs search.

Do it yourself

Crawlability checklist: make every page reachable by GoogleA crawlability checklist for a small business site: find orphan pages, robots.txt blocks, broken links, redirect chains and server errors, with free tools.

Ask us one question

One line is enough. We reply by email.

Sources

  1. Google Crawling Infrastructure: Optimize your crawl budget
  2. Google Crawling Infrastructure: Myths and facts about crawling
  3. Google Crawling Infrastructure: Improve crawling of faceted navigation URLs
  4. Google Search Central: URL structure best practices
  5. Google Search Central: How to specify a canonical URL
  6. Google Search Central: Introduction to robots.txt
  7. Google Search Central: Block Search indexing with noindex
  8. Search Console Help: Crawl Stats report
  9. Google Search Central Blog: Spring cleaning, the URL Parameters tool (March 28, 2022)