Products SEO & GEO Agent Paid Ads Agent Website Agent Conversion Agent
How it works Pricing FAQ Blog Book a call

Indexing blockers: everything that keeps pages out of Google

Updated · 8 min read

Short answer Google indexes a page only if Googlebot can reach it, the server answers with a 200 (success) code, and the page has readable content. Anything that breaks one of those blocks indexing: a noindex tag, a robots.txt rule, a login, a 4xx or lasting 5xx error, a redirect, a canonical pointing elsewhere, or content only JavaScript can show.

Want help applying this to your business? Talk it through on a free 30-minute call.

Book a call

What can block Google from indexing a page?

Google's bar has three parts. A page is eligible when "Googlebot isn't blocked", "Google receives an HTTP 200 (success) status code", and "The page has indexable content" (Google Search Central: technical requirements). Meeting all three still doesn't guarantee indexing. Failing one almost always stops it.

Googlebot is Google's crawler: the program that visits pages. A status code is the three-digit answer your server sends with every page (200 means OK, 404 means not found).

BlockerWhat it doesWhere it hides on a small business siteHow to spot it
noindex tag or headerTells Google to drop the pageA "hide from search" switch in WordPress, Wix or an SEO plugin; left on after a redesignURL Inspection names noindex as the reason
robots.txt ruleStops Google reading the pageA `Disallow: /` copied from the test siteSearch Console's robots.txt report
Login or passwordGooglebot won't crawl itMember areas, client portals, a password on a site still "in progress"Open the page in a private browser window
4xx error (404, 410, 403)Page isn't indexed; indexed pages are removedDeleted or renamed pages, a security plugin returning 403URL Inspection shows the code
5xx error or downtimeCrawling slows; pages that keep failing are droppedA struggling host, an expired plan, a broken pluginCrawl Stats report in Search Console
Soft 404Page says "not found" or is empty but answers 200Empty category pages, out-of-stock pages with no contentPage indexing report lists soft 404s
RedirectOnly the target page is indexedOld URLs forwarding to the home pageURL Inspection shows the final URL
Canonical pointing elsewhereGoogle indexes the other URL insteadA template that names the home page as canonical on every pageURL Inspection shows the canonical Google chose
Content only JavaScript showsGoogle may index an empty shellBooking widgets, menus or reviews loaded by scriptURL Inspection: view the crawled page
Removals requestHides the URL for about six monthsFiled by a former web personRemovals tool in Search Console

Sources for each row are in the sections below. Spam problems and manual actions also keep pages out; see Google manual action.

noindex vs robots.txt: the mistake that keeps pages stuck

They do different jobs, and combining them backfires. robots.txt controls crawling; noindex controls indexing. Google is blunt about robots.txt:

"It is not a mechanism for keeping a web page out of Google. To keep a web page out of Google, block indexing with noindex or password-protect the page." (Google Search Central: Introduction to robots.txt)

A page blocked in robots.txt can still show up as a bare link "without visiting the page" when other sites link to it (same page). And if you add noindex to a page robots.txt blocks, Google never reads the tag: "the crawler will never see the noindex rule, and the page can still appear in search results" (Google Search Central: Block Search indexing with noindex). Writing noindex inside robots.txt "is not supported by Google" (same page).

noindex can also hide outside the page code. The `X-Robots-Tag` HTTP header does the same job and is set on the server, so you won't see it in the page source (same page). Check for it with URL Inspection or `curl -I https://yourdomain.com/page`.

After you remove a noindex, Google has to come back to notice. For less visited pages "it may take months for Googlebot to revisit" (same page), so request indexing in URL Inspection.

Server errors and downtime: when do pages drop out?

Short outages are fine; repeated ones aren't. With 5xx errors (500, 502, 503), Google slows its crawling. Google says "already indexed URLs are preserved in the index, but eventually dropped", and URLs that "persistently return a server error" are removed (Google Search Central: HTTP status codes).

Two details catch owners out:

  • A broken robots.txt can stop the whole site. If robots.txt returns a server error, "For the first 12 hours, Google stops crawling the site" and then uses the last good copy for up to 30 days (Google Search Central: robots.txt specification). A missing robots.txt (404) is fine: Google assumes no limits (same page).
  • Maintenance mode done wrong. For a pause of one or two days, Google suggests a 503 error page. Longer than that, keep a normal home page up (Google Search Central: Temporarily pause or disable a website). Google also warns: "Don't return a 503 HTTP response status code for the robots.txt file because this blocks all crawling" (same page).

Error pages never get indexed: "Client and server error pages aren't indexed" (Google Search Central: technical requirements). A page that looks like an error but answers 200 is a soft 404: a page "telling the user that the page does not exist" with a success code, or one "with no main content" (Google Search Central: Troubleshoot crawling errors).

Does a slow website or database block indexing?

Not by itself. Google judges what its crawler receives, not how your server builds the page. Slowness becomes a blocker only when it turns into errors or timeouts. If URL Inspection shows "Hostload exceeded", Googlebot "can't crawl as many URLs from your site as it discovered" (Google Search Central: Troubleshoot crawling errors). Speed as a ranking factor is a separate question: see Core Web Vitals.

Canonical tags and duplicate pages

A canonical tag tells Google which of several similar URLs is the main one. Google treats it as advice: "indicating a canonical preference is a hint, not a rule" (Google Search Central: What is canonicalization). So a wrong canonical, such as every page naming the home page, can push your service pages out in favor of that URL.

Google lists an easy one to miss: "the demo version of the site is accidentally left accessible to crawlers" (same page). Password-protect test copies.

Don't use robots.txt or noindex to pick the main version. Google says not to use robots.txt "for canonicalization purposes", and noindex "will completely block the page from Search" (Google Search Central: How to specify a canonical URL). Use a `rel="canonical"` link, or a 301 redirect if the copy shouldn't exist.

Redirects themselves are fine. Google follows "up to 10 redirect hops" and treats a 301 as "a strong signal" that the target is the page to index (Google Search Central: HTTP status codes).

Logins, JavaScript and blocked files

  • Logins: "If a page is made private, such as requiring a log-in to view it, Googlebot will not crawl it" (Google Search Central: technical requirements).
  • JavaScript: Google renders pages in a browser before indexing, but a page with a noindex in its code is skipped: Google renders 200 pages "unless a robots meta tag or header tells Google not to index the page" (Google Search Central: JavaScript SEO basics). Menus built from script need real HTML links with an href address, or Google may miss the pages they point to (same page).
  • Blocked scripts and styles: don't block files a page needs to make sense. If their absence makes the page "harder for Google's crawler to understand", Google says "don't block them" (Google Search Central: Introduction to robots.txt).

Which pages should I block on purpose, and how?

Pick the tool by the goal (Google Search Central: Control what you share):

GoalUseWhy
Private data, client files, staging sitePasswordOnly real security; also keeps it out of Search
Thank-you, login, internal search pagesnoindexPeople can still visit; Google won't list them
Two URLs with the same contentCanonical link or 301 redirectKeeps one version; noindex would block the page entirely
Images or videos you don't want shownrobots.txtGoogle says robots.txt works for media files
Something already showing that must go nowRemovals tool, then noindex or deleteRemovals "last for about 6 months" (Google Search Central)

How to check a site for indexing blockers

  1. URL Inspection on your home page and each main service page. It shows whether the page is indexed, the status code, the canonical Google chose and the HTML Googlebot received (Google Search Central: Block Search indexing with noindex).
  2. Page indexing report and Crawl Stats report. Google says to "look at both reports" because each shows different problems (Google Search Central: technical requirements). What each status means: Search Console statuses.
  3. robots.txt report in Search Console, after any change to the file. It can request a fresh read; otherwise Google updates its copy "every 24 hours" (Google Search Central: Update your robots.txt file).
  4. Private browser window for each key page: a login prompt or error there is what Google meets too.
  5. Repeat after every redesign, host move or plugin install. Those are when blockers appear.

Example

A physiotherapy clinic after a redesign

Bookings from search fall by half. URL Inspection shows the new template sends `X-Robots-Tag: noindex` on every service page. The developer removes it; the owner requests indexing for the 8 main pages.

An electrician on cheap hosting

Crawl Stats shows server errors on most days for a month, and service pages start dropping. Moving to a more reliable host for about $30 a month ends the errors; pages return over the following weeks.

A bakery with a test site

Google shows pages from a test copy of the site at a test subdomain. The test copy had no password. Fix: password-protect it, and use the Removals tool on the test URLs.

FAQ

Does nofollow stop a page from being indexed?

No. nofollow only tells Google not to follow the links on a page or link. To keep a page out, use noindex.

Will a 404 page hurt my whole site?

No. A 404 only drops that URL. If the page moved, add a 301 redirect so visitors and Google reach the new one.

How long until a fixed page comes back?

It depends on when Google recrawls. Request indexing in URL Inspection once; for less visited pages a revisit can take weeks or longer.

Should I block Google while my new site is being built?

Use a password, not robots.txt. A password keeps both Google and visitors out, and you remove it on launch day.

Different for you?

Keep learning

NextNoindex Mistakes: How One Setting Can Hide Your Site From GoogleThe settings that add noindex by accident in WordPress, Wix, Squarespace, SEO plugins and servers, how to spot each one, and how to undo it.

Do it yourself

Crawlability checklist: make every page reachable by GoogleA crawlability checklist for a small business site: find orphan pages, robots.txt blocks, broken links, redirect chains and server errors, with free tools.

Ask us one question

One line is enough. We reply by email.

Sources

  1. Google Search Central: Google Search technical requirements
  2. Google Search Central: Block Search indexing with noindex
  3. Google Search Central: Introduction to robots.txt
  4. Google Search Central: How Google interprets the robots.txt specification
  5. Google Search Central: How HTTP status codes affect Google's crawlers
  6. Google Search Central: Temporarily pause or disable a website
  7. Google Search Central: What is canonicalization
  8. Google Search Central: How to specify a canonical URL
  9. Google Search Central: Understand JavaScript SEO basics
  10. Google Search Central: Troubleshoot Google Search crawling errors
  11. Google Search Central: Control what you share with Google Search
  12. Google Search Central: Remove a page hosted on your site from Google
  13. Google Search Central: Update your robots.txt file