Products SEO & GEO Agent Paid Ads Agent Website Agent Conversion Agent
How it works Pricing FAQ Blog Book a call

Crawlability checklist: make every page reachable by Google

Updated · 8 min read

Short answer Google reaches pages by following links, so crawlability comes down to three checks: every page you want found has a normal link from another page on your site, robots.txt doesn't block it or the files it needs, and the link lands on a working page, not a 404, a server error or a long redirect chain. Search Console shows all three free.

Want help applying this to your business? Talk it through on a free 30-minute call.

Book a call

Crawlability checklist: the checks that matter

Crawling is Google's robot (Googlebot) visiting your pages. A page it can't reach can't be indexed, so it can't show up at all. Google says it "doesn't crawl all the pages it discovered": some are blocked by the owner, some sit behind a login, and server or network trouble stops others (Google Search Central: How Search works).

Run these in order. Each takes minutes in free tools.

#CheckWhere to lookFix
1Every page you want found is linked from another page on your siteSearch Console → Links → Top internally linked pagesAdd a link from a related page or the menu
2Links are real HTML linksRight-click a menu item → Inspect: the code should be an "a" tag with an "href" addressAsk your web person to use standard links
3robots.txt doesn't block pages you want foundyourdomain.com/robots.txt, then URL Inspection → "Crawl allowed?"Delete the rule that blocks them
4robots.txt doesn't block CSS, JavaScript or images the page needsURL Inspection → live test → View tested pageUnblock those files
5No links to dead pages (404)Page indexing report → "Not found (404)"Fix the link, or redirect the old address
6No redirect chains or loopsPage indexing report → "Redirect error"Point each old address straight to the final page
7No server errors (5xx)Page indexing report → "Server error (5xx)"Ask your host; recurring errors need a fix, not a retry
8The mobile version has the same pages and linksOpen the site on your phone and use the menuPut every link from the desktop menu in the mobile menu

Search Console is Google's free dashboard for your site. If it isn't set up, see Google Search Console for a small business.

How do I find orphan pages?

An orphan page is one no other page on your site links to. Google can still learn of it from a sitemap or another site, but it treats your links as the main road in. Its rule:

"Every page you care about should have a link from at least one other page on your site." (Google Search Central: Link best practices)

To find them:

  1. List the pages you want customers to find: each service, each area you serve, contact, about.
  2. In Search Console, open Links → Top internally linked pages. Google suggests this table to "confirm that the core site pages (home page, contact page) are properly linked within your site" (Search Console Help: Links report).
  3. Any page from step 1 missing from the table, or with one or two links, is your fix list. The report is a sample, not every link (same page), so open the page's address in URL Inspection to confirm.
  4. Link it from the page a customer would come from: the service page from the home page and menu, the area page from the service page.

Common orphans on small sites: landing pages built for ads, old blog posts dropped from the blog list, and pages a website builder hides from the menu.

Check the link type too. Google can generally crawl a link only if it is a standard HTML link (an "a" element with an href address); buttons and menus that change pages only through a script may not be followed (Link best practices).

Is robots.txt blocking pages I need?

robots.txt is a small text file at yourdomain.com/robots.txt that tells crawlers which addresses to stay out of. Open it in your browser. A line `Disallow: /` under `User-agent: *` blocks the whole site; `Disallow: /services/` blocks everything in that folder.

Three things to know:

  • No file is fine. Google's robots.txt report says not having one "means that Google can crawl all URLs on your site" (Search Console Help: robots.txt report). A broken file is worse: if the address returns a server error, Google stops crawling the site for the first 12 hours (Google Crawling Infrastructure: How Google interprets the robots.txt specification).
  • It doesn't hide pages. Google is direct:
"it is not a mechanism for keeping a web page out of Google. To keep a web page out of Google, block indexing with noindex or password-protect the page." (Google Search Central: Introduction to robots.txt)
  • Don't block the files a page needs. If missing image, script or style files make a page harder to understand, "don't block them" (same page).

To test one page, paste its address into URL Inspection in Search Console. The "Crawl allowed?" line says whether robots.txt lets Google in (Search Console Help: URL Inspection). On Wix or Blogger you may not edit the file at all; the builder's search settings control it (Introduction to robots.txt).

Blocking AI crawlers like GPTBot is a separate choice. See should you block AI crawlers?

Open Search Console → Pages (the Page indexing report) and read the reasons under "Why pages aren't indexed" (Search Console Help: Page indexing report).

Reason in the reportWhat it meansWhat to do
Not found (404)The address doesn't existFix only those "you link to yourself or list in a sitemap"; for moved pages, redirect to the new address (same page)
Soft 404The page says "not found" or is near empty but tells Google it's fineReturn a real 404, or add real content
Redirect errorA chain too long, a loop, or a bad address in the chainRedirect old addresses straight to the final page
Server error (5xx)Your server failed when Google askedAsk your host to find the cause
Blocked by robots.txtA robots.txt rule stops GoogleRemove the rule if you want the page found

Why each matters:

  • 404s: a 404 page drops out of the index and Google visits it less over time (Google Crawling Infrastructure: HTTP status codes). The cost is the customer who clicks your own dead link.
  • Redirects: Googlebot follows up to 10 hops (same page), but Google advises sending each old address "to the final destination directly", and if not, "ideally no more than 3 and fewer than 5" hops (Google Search Central: Site moves). Each redesign can add a hop. See traffic dropped after a website redesign.
  • Server errors make Google "temporarily slow down with crawling", and pages already indexed "are preserved in the index, but eventually dropped" (Google Crawling Infrastructure: HTTP status codes). One bad night is fine. Errors every week are a hosting problem.

For a full list of broken links on your own pages, a desktop crawler helps. See the Screaming Frog walkthrough; the same tool crawls your own site.

Do I need a sitemap, and should I worry about crawl budget?

A sitemap is a file listing your pages for search engines. Google says you might not need one if your site has "about 500 pages or fewer", is well linked so Googlebot can reach every important page from the home page, and has few videos, images or news pages you want in search (Google Search Central: What is a sitemap). Most builders create one anyway; it does no harm.

Crawl budget is how many pages Google will crawl on your site in a given time. Google's guide is written for sites with "1 million+ unique pages" that change weekly, or 10,000+ that change daily, and for everyone else says keeping the sitemap current and checking the Page indexing report "is adequate" (Google Crawling Infrastructure: Crawl budget management). A 30-page service site can skip it.

Mobile counts first. Google indexes the mobile version, and "only the content shown on the mobile site is used for indexing" (Google Search Central: Mobile-first indexing). A service page linked only from the desktop menu can turn into an orphan on mobile.

Structured data (schema) and page speed affect how pages appear and rank, not whether Google can reach them. Canonical tags pick which of several duplicate pages gets indexed; they don't help Google reach a page either (Google Search Central: What is canonicalization). If a page is reached but still not indexed, that is a different problem: see crawled or discovered, currently not indexed.

Example

A cleaner with five area pages

A cleaning company built pages for five towns, but the menu links only to "Services". Search Console's Links report shows the home page and contact page, and none of the town pages. Adding an "Areas we serve" block on the services page, linking all five, fixes the orphans in an afternoon.

A clinic after a redesign

A clinic moved /services/botox.html to /botox/ in 2024, then to /treatments/botox/ in 2026, but an old rule still sends /treatments/botox/ back to /botox/. The Page indexing report lists it under "Redirect error": a loop. Deleting the old rule and pointing every old address straight to /treatments/botox/ fixes it.

A contractor's blocked folder

A contractor's robots.txt, copied from a template, has `Disallow: /wp-content/`. URL Inspection's screenshot shows the page without styling or photos. Deleting that line lets Google load the page as customers see it.

FAQ

What is crawlability in SEO?

It's whether a search engine's robot can reach a page by following links and load it. A page it can't reach can't be indexed or ranked.

Is crawling the same as indexing?

No. Crawling is Google visiting the page; indexing is Google storing it so it can show in results. A page can be crawled and still not indexed.

How do I check if Google can crawl a page?

Paste the address into URL Inspection in Search Console and run a live test. "Crawl allowed?" and "Page fetch" tell you if robots.txt or the server stopped Google (Search Console Help: URL Inspection).

Do broken links hurt my rankings?

Google describes no sitewide penalty. A 404 page drops out of the index, and visitors who hit it leave, so fix the ones you link to.

How often should I run this checklist?

After any redesign, move or new set of pages, and a quick look at the Page indexing report once a month.

Different for you?

Keep learning

NextSitemap best practices today: lastmod, IndexNow and AI botsA current XML sitemap checklist: what Google and Bing read, why only lastmod counts, how IndexNow reaches Bing fast, and what AI search bots need.

Do it yourself

Core Web Vitals Checklist: Quick Wins for a Faster SiteA one-week Core Web Vitals checklist for a small business site: find the failing metric, then the image, layout and script fixes an owner can make.

Ask us one question

One line is enough. We reply by email.

Sources

  1. Google Search Central: Link best practices for Google
  2. Google Search Central: In-depth guide to how Google Search works
  3. Google Crawling Infrastructure: How Google interprets the robots.txt specification
  4. Google Search Central: Introduction to robots.txt
  5. Google Crawling Infrastructure: How HTTP status codes affect Google's crawlers
  6. Google Search Central: Site moves with URL changes
  7. Google Search Central: What is a sitemap
  8. Google Crawling Infrastructure: Crawl budget management
  9. Google Search Central: Mobile-first indexing best practices
  10. Google Search Central: What is canonicalization
  11. Search Console Help: Page indexing report
  12. Search Console Help: robots.txt report
  13. Search Console Help: URL Inspection tool
  14. Search Console Help: Links report