Crawl efficiency: does it really grow organic traffic?
Short answer Rarely. Google says crawling is not a ranking signal, and its crawl budget guide is for sites with roughly 10,000+ pages that change daily or 1 million+ pages. If your new pages get crawled soon after publishing, crawl efficiency isn't what holds back your traffic. Fix server errors and duplicate URLs, then put your time into better pages.
Want help applying this to your business? Talk it through on a free 30-minute call.
Book a callDoes crawl efficiency grow organic traffic?
Not by itself. Crawling is how Google's bot (Googlebot) reads your pages. A page must be crawled to show up in Google, but crawling more often doesn't lift it in the results. Google's crawling team said so directly:
"An increased crawl rate will not necessarily lead to better positions in Search results." (Google Search Central Blog: What crawl budget means for Googlebot)
The same post adds that crawling "is not a ranking signal". Being crawled is the ticket in the door. What ranks you is still whether your page answers the search better than the others.
Crawl efficiency means Google spends its visits on your pages that matter, not on duplicates, errors and dead ends. Crawl budget is how many of your URLs Google can and wants to crawl. Google sets it from two things: how much load your server handles well (crawl capacity), and how much Google wants your pages (crawl demand), based on popularity and how often they change.
When does crawl efficiency matter?
When the site is big or changes fast. Google's crawl budget guide opens with a test:
"If your site doesn't have a large number of pages that change rapidly, or if your pages seem to be crawled the same day that they are published, you don't need to read this guide." (Google Crawling Infrastructure: Optimize your crawl budget)
It names who the guide is for, and calls the numbers "rough" estimates, not thresholds:
| Site | Does crawl budget matter? |
|---|---|
| 1 million+ pages that change about weekly | Yes |
| 10,000+ pages that change daily | Yes |
| Many URLs marked "Discovered - currently not indexed" in Search Console | Yes, look into it |
| Fewer than a few thousand pages, like most service business sites | Rarely; usually crawled efficiently (2017 post) |
For everyone else, Google says keeping your sitemap up to date and checking the Page indexing report "is adequate". The last row comes from its 2017 post: a site with "fewer than a few thousand URLs" is usually crawled efficiently.
How do I check whether Google crawls my site well?
Three free checks in Google Search Console take ten minutes:
- URL Inspection. Paste in a page you published last week. If it says the page is on Google and shows a recent crawl date, crawling is fine.
- Page indexing report. Look at the reasons pages aren't indexed. A few is normal. A large share under "Discovered - currently not indexed" is the sign Google's guide names. See crawled or discovered, currently not indexed.
- Crawl Stats report. It shows total crawl requests, average response time and host status, meaning whether Google had trouble reaching your server in the last 90 days (Google Search Central Blog, 2020). A host status problem is worth fixing at any site size.
New to Search Console? Start with Google Search Console for a small business.
What wastes Google's crawling, and what's worth fixing?
On a small site, fix the items below because they hurt visitors too, not to save budget. Google's crawl budget guide lists them as good practice:
| Problem | Why it matters | Fix |
|---|---|---|
| Server errors (5xx) or "too many requests" (429) | Google lowers its crawl limit and crawls less | Ask your host; upgrade a plan that keeps timing out |
| Duplicate URLs: sorted views, filter and tag pages | Google tries to crawl every URL it knows | Merge them, or block the junk ones in robots.txt |
| Soft 404s (an empty page that says "not found" but returns OK) | Google keeps crawling them | Return a real 404 or 410, or add the content back |
| Long redirect chains | Google says they hurt crawling; its crawlers follow up to 10 hops | Point old URLs straight to the final page |
| Slow pages | Faster pages let Google read more | See Core Web Vitals |
One trap: don't add a noindex tag to "save" crawling. Google says it still requests those pages and then drops them, "wasting crawling time". And blocking pages in robots.txt doesn't hand their share to other pages unless Google is already at your server's limit.
Are there target numbers, like a response time or error rate?
Not from Google. You'll see figures online such as a 300-millisecond server response or under 1% server errors. Google's crawl budget guide gives no such targets. It says a site whose response times stay stable or improve gets more crawling, and one that slows down or throws server errors gets less. Watch the trend in Crawl Stats, not a magic number.
Does a small site need an XML sitemap?
Usually it helps, but it's optional for small sites. Google defines small as "about 500 pages or fewer" and says such a site may not need one if every page is linked from the menu or other pages (Google Search Central: Learn about sitemaps). WordPress, Wix and similar builders usually create one for you. Submit it in Search Console once and leave it.
Example
Illustrative cases with invented numbers.
A plumber with 30 pages (US dollars)
New service pages show up in URL Inspection within two days. An SEO offers a $900 "crawl budget audit". Skip it. Crawling isn't the bottleneck; the money does more on better service pages or review requests.
A landscaper near Toronto after a redesign
Old URLs redirect to a temporary page, then to the new page. Crawl Stats shows many redirect responses. Point each old URL straight to its new page in one hop. For traffic losses after a rebuild, see the redesign guide linked below.
An online parts store with 40,000 URLs
Filters for size, color and brand create endless URL combinations, and Search Console lists 25,000 as "Discovered - currently not indexed". Here crawl efficiency matters: block the filter combinations nobody searches for and keep one clean URL per product.
FAQ
Can I ask Google to crawl my site more often?
Not directly. Google gives two ways to earn more crawling: more server capacity if your server is the limit, and content that is more popular, useful and unique.
Does the crawl-delay line in robots.txt work on Google?
No. Google's 2017 crawl budget post says Googlebot does not process the crawl-delay rule.
Do images, CSS and JavaScript files use crawl budget?
Yes. Google says any URL it crawls counts, including embedded files. On a small site this rarely adds up to a problem.
Do pages blocked in robots.txt use crawl budget?
No. Google's 2017 post says disallowed URLs do not affect crawl budget.
Different for you?
- Your pages are crawled but not indexed: crawled or discovered, currently not indexed.
- Your site doesn't show on Google at all: website not showing on Google.
- You run a store with thousands of product URLs: ecommerce SEO.
- Traffic fell after a new website: traffic dropped after a redesign.
- You'd rather have the technical checks done for you: SEO & GEO Agent.