Crawlability checklist: make every page reachable by Google
Short answer Google reaches pages by following links, so crawlability comes down to three checks: every page you want found has a normal link from another page on your site, robots.txt doesn't block it or the files it needs, and the link lands on a working page, not a 404, a server error or a long redirect chain. Search Console shows all three free.
Want help applying this to your business? Talk it through on a free 30-minute call.
Book a callCrawlability checklist: the checks that matter
Crawling is Google's robot (Googlebot) visiting your pages. A page it can't reach can't be indexed, so it can't show up at all. Google says it "doesn't crawl all the pages it discovered": some are blocked by the owner, some sit behind a login, and server or network trouble stops others (Google Search Central: How Search works).
Run these in order. Each takes minutes in free tools.
| # | Check | Where to look | Fix |
|---|---|---|---|
| 1 | Every page you want found is linked from another page on your site | Search Console → Links → Top internally linked pages | Add a link from a related page or the menu |
| 2 | Links are real HTML links | Right-click a menu item → Inspect: the code should be an "a" tag with an "href" address | Ask your web person to use standard links |
| 3 | robots.txt doesn't block pages you want found | yourdomain.com/robots.txt, then URL Inspection → "Crawl allowed?" | Delete the rule that blocks them |
| 4 | robots.txt doesn't block CSS, JavaScript or images the page needs | URL Inspection → live test → View tested page | Unblock those files |
| 5 | No links to dead pages (404) | Page indexing report → "Not found (404)" | Fix the link, or redirect the old address |
| 6 | No redirect chains or loops | Page indexing report → "Redirect error" | Point each old address straight to the final page |
| 7 | No server errors (5xx) | Page indexing report → "Server error (5xx)" | Ask your host; recurring errors need a fix, not a retry |
| 8 | The mobile version has the same pages and links | Open the site on your phone and use the menu | Put every link from the desktop menu in the mobile menu |
Search Console is Google's free dashboard for your site. If it isn't set up, see Google Search Console for a small business.
How do I find orphan pages?
An orphan page is one no other page on your site links to. Google can still learn of it from a sitemap or another site, but it treats your links as the main road in. Its rule:
"Every page you care about should have a link from at least one other page on your site." (Google Search Central: Link best practices)
To find them:
- List the pages you want customers to find: each service, each area you serve, contact, about.
- In Search Console, open Links → Top internally linked pages. Google suggests this table to "confirm that the core site pages (home page, contact page) are properly linked within your site" (Search Console Help: Links report).
- Any page from step 1 missing from the table, or with one or two links, is your fix list. The report is a sample, not every link (same page), so open the page's address in URL Inspection to confirm.
- Link it from the page a customer would come from: the service page from the home page and menu, the area page from the service page.
Common orphans on small sites: landing pages built for ads, old blog posts dropped from the blog list, and pages a website builder hides from the menu.
Check the link type too. Google can generally crawl a link only if it is a standard HTML link (an "a" element with an href address); buttons and menus that change pages only through a script may not be followed (Link best practices).
Is robots.txt blocking pages I need?
robots.txt is a small text file at yourdomain.com/robots.txt that tells crawlers which addresses to stay out of. Open it in your browser. A line `Disallow: /` under `User-agent: *` blocks the whole site; `Disallow: /services/` blocks everything in that folder.
Three things to know:
- No file is fine. Google's robots.txt report says not having one "means that Google can crawl all URLs on your site" (Search Console Help: robots.txt report). A broken file is worse: if the address returns a server error, Google stops crawling the site for the first 12 hours (Google Crawling Infrastructure: How Google interprets the robots.txt specification).
- It doesn't hide pages. Google is direct:
"it is not a mechanism for keeping a web page out of Google. To keep a web page out of Google, block indexing with noindex or password-protect the page." (Google Search Central: Introduction to robots.txt)
- Don't block the files a page needs. If missing image, script or style files make a page harder to understand, "don't block them" (same page).
To test one page, paste its address into URL Inspection in Search Console. The "Crawl allowed?" line says whether robots.txt lets Google in (Search Console Help: URL Inspection). On Wix or Blogger you may not edit the file at all; the builder's search settings control it (Introduction to robots.txt).
Blocking AI crawlers like GPTBot is a separate choice. See should you block AI crawlers?
Broken links, redirects and server errors
Open Search Console → Pages (the Page indexing report) and read the reasons under "Why pages aren't indexed" (Search Console Help: Page indexing report).
| Reason in the report | What it means | What to do |
|---|---|---|
| Not found (404) | The address doesn't exist | Fix only those "you link to yourself or list in a sitemap"; for moved pages, redirect to the new address (same page) |
| Soft 404 | The page says "not found" or is near empty but tells Google it's fine | Return a real 404, or add real content |
| Redirect error | A chain too long, a loop, or a bad address in the chain | Redirect old addresses straight to the final page |
| Server error (5xx) | Your server failed when Google asked | Ask your host to find the cause |
| Blocked by robots.txt | A robots.txt rule stops Google | Remove the rule if you want the page found |
Why each matters:
- 404s: a 404 page drops out of the index and Google visits it less over time (Google Crawling Infrastructure: HTTP status codes). The cost is the customer who clicks your own dead link.
- Redirects: Googlebot follows up to 10 hops (same page), but Google advises sending each old address "to the final destination directly", and if not, "ideally no more than 3 and fewer than 5" hops (Google Search Central: Site moves). Each redesign can add a hop. See traffic dropped after a website redesign.
- Server errors make Google "temporarily slow down with crawling", and pages already indexed "are preserved in the index, but eventually dropped" (Google Crawling Infrastructure: HTTP status codes). One bad night is fine. Errors every week are a hosting problem.
For a full list of broken links on your own pages, a desktop crawler helps. See the Screaming Frog walkthrough; the same tool crawls your own site.
Do I need a sitemap, and should I worry about crawl budget?
A sitemap is a file listing your pages for search engines. Google says you might not need one if your site has "about 500 pages or fewer", is well linked so Googlebot can reach every important page from the home page, and has few videos, images or news pages you want in search (Google Search Central: What is a sitemap). Most builders create one anyway; it does no harm.
Crawl budget is how many pages Google will crawl on your site in a given time. Google's guide is written for sites with "1 million+ unique pages" that change weekly, or 10,000+ that change daily, and for everyone else says keeping the sitemap current and checking the Page indexing report "is adequate" (Google Crawling Infrastructure: Crawl budget management). A 30-page service site can skip it.
Mobile counts first. Google indexes the mobile version, and "only the content shown on the mobile site is used for indexing" (Google Search Central: Mobile-first indexing). A service page linked only from the desktop menu can turn into an orphan on mobile.
Structured data (schema) and page speed affect how pages appear and rank, not whether Google can reach them. Canonical tags pick which of several duplicate pages gets indexed; they don't help Google reach a page either (Google Search Central: What is canonicalization). If a page is reached but still not indexed, that is a different problem: see crawled or discovered, currently not indexed.
Example
A cleaner with five area pages
A cleaning company built pages for five towns, but the menu links only to "Services". Search Console's Links report shows the home page and contact page, and none of the town pages. Adding an "Areas we serve" block on the services page, linking all five, fixes the orphans in an afternoon.
A clinic after a redesign
A clinic moved /services/botox.html to /botox/ in 2024, then to /treatments/botox/ in 2026, but an old rule still sends /treatments/botox/ back to /botox/. The Page indexing report lists it under "Redirect error": a loop. Deleting the old rule and pointing every old address straight to /treatments/botox/ fixes it.
A contractor's blocked folder
A contractor's robots.txt, copied from a template, has `Disallow: /wp-content/`. URL Inspection's screenshot shows the page without styling or photos. Deleting that line lets Google load the page as customers see it.
FAQ
What is crawlability in SEO?
It's whether a search engine's robot can reach a page by following links and load it. A page it can't reach can't be indexed or ranked.
Is crawling the same as indexing?
No. Crawling is Google visiting the page; indexing is Google storing it so it can show in results. A page can be crawled and still not indexed.
How do I check if Google can crawl a page?
Paste the address into URL Inspection in Search Console and run a live test. "Crawl allowed?" and "Page fetch" tell you if robots.txt or the server stopped Google (Search Console Help: URL Inspection).
Do broken links hurt my rankings?
Google describes no sitewide penalty. A 404 page drops out of the index, and visitors who hit it leave, so fix the ones you link to.
How often should I run this checklist?
After any redesign, move or new set of pages, and a quick look at the Page indexing report once a month.
Different for you?
- Your whole site is missing from Google: start with website not showing up on Google.
- Pages are reached but not indexed: see crawled or discovered, currently not indexed.
- You just redesigned or changed domains: see traffic dropped after a website redesign.
- The menu or page structure needs rebuilding: that's website work. See /website/.
- You want the checks run every month for you: see /seo/.