Indexing blockers: everything that keeps pages out of Google
Short answer Google indexes a page only if Googlebot can reach it, the server answers with a 200 (success) code, and the page has readable content. Anything that breaks one of those blocks indexing: a noindex tag, a robots.txt rule, a login, a 4xx or lasting 5xx error, a redirect, a canonical pointing elsewhere, or content only JavaScript can show.
Want help applying this to your business? Talk it through on a free 30-minute call.
Book a callWhat can block Google from indexing a page?
Google's bar has three parts. A page is eligible when "Googlebot isn't blocked", "Google receives an HTTP 200 (success) status code", and "The page has indexable content" (Google Search Central: technical requirements). Meeting all three still doesn't guarantee indexing. Failing one almost always stops it.
Googlebot is Google's crawler: the program that visits pages. A status code is the three-digit answer your server sends with every page (200 means OK, 404 means not found).
| Blocker | What it does | Where it hides on a small business site | How to spot it |
|---|---|---|---|
| noindex tag or header | Tells Google to drop the page | A "hide from search" switch in WordPress, Wix or an SEO plugin; left on after a redesign | URL Inspection names noindex as the reason |
| robots.txt rule | Stops Google reading the page | A `Disallow: /` copied from the test site | Search Console's robots.txt report |
| Login or password | Googlebot won't crawl it | Member areas, client portals, a password on a site still "in progress" | Open the page in a private browser window |
| 4xx error (404, 410, 403) | Page isn't indexed; indexed pages are removed | Deleted or renamed pages, a security plugin returning 403 | URL Inspection shows the code |
| 5xx error or downtime | Crawling slows; pages that keep failing are dropped | A struggling host, an expired plan, a broken plugin | Crawl Stats report in Search Console |
| Soft 404 | Page says "not found" or is empty but answers 200 | Empty category pages, out-of-stock pages with no content | Page indexing report lists soft 404s |
| Redirect | Only the target page is indexed | Old URLs forwarding to the home page | URL Inspection shows the final URL |
| Canonical pointing elsewhere | Google indexes the other URL instead | A template that names the home page as canonical on every page | URL Inspection shows the canonical Google chose |
| Content only JavaScript shows | Google may index an empty shell | Booking widgets, menus or reviews loaded by script | URL Inspection: view the crawled page |
| Removals request | Hides the URL for about six months | Filed by a former web person | Removals tool in Search Console |
Sources for each row are in the sections below. Spam problems and manual actions also keep pages out; see Google manual action.
noindex vs robots.txt: the mistake that keeps pages stuck
They do different jobs, and combining them backfires. robots.txt controls crawling; noindex controls indexing. Google is blunt about robots.txt:
"It is not a mechanism for keeping a web page out of Google. To keep a web page out of Google, block indexing with noindex or password-protect the page." (Google Search Central: Introduction to robots.txt)
A page blocked in robots.txt can still show up as a bare link "without visiting the page" when other sites link to it (same page). And if you add noindex to a page robots.txt blocks, Google never reads the tag: "the crawler will never see the noindex rule, and the page can still appear in search results" (Google Search Central: Block Search indexing with noindex). Writing noindex inside robots.txt "is not supported by Google" (same page).
noindex can also hide outside the page code. The `X-Robots-Tag` HTTP header does the same job and is set on the server, so you won't see it in the page source (same page). Check for it with URL Inspection or `curl -I https://yourdomain.com/page`.
After you remove a noindex, Google has to come back to notice. For less visited pages "it may take months for Googlebot to revisit" (same page), so request indexing in URL Inspection.
Server errors and downtime: when do pages drop out?
Short outages are fine; repeated ones aren't. With 5xx errors (500, 502, 503), Google slows its crawling. Google says "already indexed URLs are preserved in the index, but eventually dropped", and URLs that "persistently return a server error" are removed (Google Search Central: HTTP status codes).
Two details catch owners out:
- A broken robots.txt can stop the whole site. If robots.txt returns a server error, "For the first 12 hours, Google stops crawling the site" and then uses the last good copy for up to 30 days (Google Search Central: robots.txt specification). A missing robots.txt (404) is fine: Google assumes no limits (same page).
- Maintenance mode done wrong. For a pause of one or two days, Google suggests a 503 error page. Longer than that, keep a normal home page up (Google Search Central: Temporarily pause or disable a website). Google also warns: "Don't return a 503 HTTP response status code for the robots.txt file because this blocks all crawling" (same page).
Error pages never get indexed: "Client and server error pages aren't indexed" (Google Search Central: technical requirements). A page that looks like an error but answers 200 is a soft 404: a page "telling the user that the page does not exist" with a success code, or one "with no main content" (Google Search Central: Troubleshoot crawling errors).
Does a slow website or database block indexing?
Not by itself. Google judges what its crawler receives, not how your server builds the page. Slowness becomes a blocker only when it turns into errors or timeouts. If URL Inspection shows "Hostload exceeded", Googlebot "can't crawl as many URLs from your site as it discovered" (Google Search Central: Troubleshoot crawling errors). Speed as a ranking factor is a separate question: see Core Web Vitals.
Canonical tags and duplicate pages
A canonical tag tells Google which of several similar URLs is the main one. Google treats it as advice: "indicating a canonical preference is a hint, not a rule" (Google Search Central: What is canonicalization). So a wrong canonical, such as every page naming the home page, can push your service pages out in favor of that URL.
Google lists an easy one to miss: "the demo version of the site is accidentally left accessible to crawlers" (same page). Password-protect test copies.
Don't use robots.txt or noindex to pick the main version. Google says not to use robots.txt "for canonicalization purposes", and noindex "will completely block the page from Search" (Google Search Central: How to specify a canonical URL). Use a `rel="canonical"` link, or a 301 redirect if the copy shouldn't exist.
Redirects themselves are fine. Google follows "up to 10 redirect hops" and treats a 301 as "a strong signal" that the target is the page to index (Google Search Central: HTTP status codes).
Logins, JavaScript and blocked files
- Logins: "If a page is made private, such as requiring a log-in to view it, Googlebot will not crawl it" (Google Search Central: technical requirements).
- JavaScript: Google renders pages in a browser before indexing, but a page with a noindex in its code is skipped: Google renders 200 pages "unless a robots meta tag or header tells Google not to index the page" (Google Search Central: JavaScript SEO basics). Menus built from script need real HTML links with an href address, or Google may miss the pages they point to (same page).
- Blocked scripts and styles: don't block files a page needs to make sense. If their absence makes the page "harder for Google's crawler to understand", Google says "don't block them" (Google Search Central: Introduction to robots.txt).
Which pages should I block on purpose, and how?
Pick the tool by the goal (Google Search Central: Control what you share):
| Goal | Use | Why |
|---|---|---|
| Private data, client files, staging site | Password | Only real security; also keeps it out of Search |
| Thank-you, login, internal search pages | noindex | People can still visit; Google won't list them |
| Two URLs with the same content | Canonical link or 301 redirect | Keeps one version; noindex would block the page entirely |
| Images or videos you don't want shown | robots.txt | Google says robots.txt works for media files |
| Something already showing that must go now | Removals tool, then noindex or delete | Removals "last for about 6 months" (Google Search Central) |
How to check a site for indexing blockers
- URL Inspection on your home page and each main service page. It shows whether the page is indexed, the status code, the canonical Google chose and the HTML Googlebot received (Google Search Central: Block Search indexing with noindex).
- Page indexing report and Crawl Stats report. Google says to "look at both reports" because each shows different problems (Google Search Central: technical requirements). What each status means: Search Console statuses.
- robots.txt report in Search Console, after any change to the file. It can request a fresh read; otherwise Google updates its copy "every 24 hours" (Google Search Central: Update your robots.txt file).
- Private browser window for each key page: a login prompt or error there is what Google meets too.
- Repeat after every redesign, host move or plugin install. Those are when blockers appear.
Example
A physiotherapy clinic after a redesign
Bookings from search fall by half. URL Inspection shows the new template sends `X-Robots-Tag: noindex` on every service page. The developer removes it; the owner requests indexing for the 8 main pages.
An electrician on cheap hosting
Crawl Stats shows server errors on most days for a month, and service pages start dropping. Moving to a more reliable host for about $30 a month ends the errors; pages return over the following weeks.
A bakery with a test site
Google shows pages from a test copy of the site at a test subdomain. The test copy had no password. Fix: password-protect it, and use the Removals tool on the test URLs.
FAQ
Does nofollow stop a page from being indexed?
No. nofollow only tells Google not to follow the links on a page or link. To keep a page out, use noindex.
Will a 404 page hurt my whole site?
No. A 404 only drops that URL. If the page moved, add a 301 redirect so visitors and Google reach the new one.
How long until a fixed page comes back?
It depends on when Google recrawls. Request indexing in URL Inspection once; for less visited pages a revisit can take weeks or longer.
Should I block Google while my new site is being built?
Use a password, not robots.txt. A password keeps both Google and visitors out, and you remove it on launch day.
Different for you?
- The whole site is missing, not one page: website not showing up on Google.
- Search Console says "Crawled - currently not indexed": what each status means.
- New to Search Console: the reports a small business should read.
- Pages fell out after a redesign: traffic dropped after a redesign.
- You want to block AI crawlers but not Google: GPTBot, ClaudeBot and robots.txt.
- You'd rather have someone find and fix blockers: SEO & GEO Agent.