Products SEO & GEO Agent Paid Ads Agent Website Agent Conversion Agent
How it works Pricing FAQ Blog Book a call

Check for noindex tags across your whole site

Updated · 6 min read

Short answer Open Search Console's Page indexing report and look for the reason "Excluded by 'noindex' tag". If pages you want found are listed, something is telling Google to skip them. Then crawl the site with a free crawler to catch every page, and check one page's HTTP header, where a noindex can hide outside the page code.

Want help applying this to your business? Talk it through on a free 30-minute call.

Book a call

How do I check my whole site for noindex?

A noindex is a line in a page's code, or in the server's reply, that tells Google to leave the page out of search. Google describes what happens next:

"When Googlebot crawls that page and extracts the tag or header, Google will drop that page entirely from Google Search results, regardless of whether other sites link to it." (Google Search Central: Block Search indexing with noindex)

One page by mistake is a small problem. A setting that puts it on every page can empty your search traffic. Run three checks, in this order:

CheckWhat it catchesCost
1. Search Console, Indexing > PagesEvery page where Google found a noindexFree
2. A site crawlerEvery page that has one now, including pages Google hasn't revisitedFree up to 500 URLs
3. Your home page's code and HTTP headerA noindex set by the server, invisible in the page editorFree

A `site:yourdomain.com` search on Google is a quick first look, but Google says its results are "not always exhaustive". It shows what is in, not why something is missing.

Step 1: read the noindex list in Search Console

In Search Console, open Indexing > Pages. Under "Why pages aren't indexed", find "Excluded by 'noindex' tag" and click it to see example pages. The list shows up to 1,000 examples (Search Console Help: Page indexing report).

Read it as two piles:

  • Fine to keep: thank-you pages, login, cart and checkout, internal search results, tag archives.
  • A problem: your home page, service pages, location pages, blog posts.

If the problem pile is long, look for a pattern. All blog posts means a content-type setting. Every page means a sitewide switch. Search Console's help also flags the warning sign: a drop in indexed pages without new errors can mean you are blocking pages "via robots.txt, 'noindex' or a required login".

This list only shows what Google saw on its last visit. Google says that can take a while:

"Depending on the importance of the page on the internet, it may take months for Googlebot to revisit a page." (Google Search Central: Block Search indexing with noindex)

So a noindex added last week may not appear yet. That is why step 2 matters.

Step 2: crawl the site to see every page today

A crawler is a program that visits every page on your site the way Google does and lists what it finds. Screaming Frog SEO Spider crawls up to 500 URLs free, enough for most service business sites; a paid yearly licence removes the limit (Screaming Frog). It reports noindex found in either the page code or the HTTP header.

  1. Download it, paste your home page address, and start the crawl.
  2. Open the Directives tab and filter to noindex.
  3. Export the list and compare it with the "fine to keep" pile above.

Run it again after every launch, theme change or SEO plugin update. It takes minutes on a small site.

Step 3: check the HTTP header, not just the page

Google accepts noindex in two places: a robots meta tag in the page code (it contains `name="robots" content="noindex"`) or an HTTP header the server sends with the page (`X-Robots-Tag: noindex`). The header never shows in your page editor or "View page source".

To check it, open your home page in Chrome, right-click, choose Inspect, open the Network tab and reload. Click the first row (your page) and read the response headers. Search Console's help says the same: search "the page source or response headers for the word 'noindex'". A header noindex usually comes from hosting, a security plugin or a CDN rule, not your website builder.

For one page, Search Console's URL Inspection tool settles it. Run the live test and read "Indexing allowed?". "No" means a noindex is still there.

Where does a sitewide noindex come from?

Most come from a switch someone flipped on purpose, then forgot.

SourceWhat happensWhere to look
WordPress "Discourage search engines"Adds a noindex tag to every page. Often left on from building the site, or copied over from a staging (test) siteSettings > Reading (WordPress Documentation)
SEO plugin content typesYoast's toggle per type (Posts, Pages, Products) noindexes the whole type when offYoast SEO > Settings > Content types (Yoast)
Plugin changeA new SEO plugin brings its own defaults, or two plugins disagreePlugins list; keep one SEO plugin
WixThe "Let search engines index your site" toggle is offSEO Dashboard > Go to SEO Settings (Wix Help Center)
ShopifyPrivate mode hides "all of your online store's pages" from visitors and search enginesOnline Store > Preferences > Private mode (Shopify Help Center)
Server or CDNAn `X-Robots-Tag` header added for a test site and pushed to liveAsk your host or developer
Theme or templateOne template carries a noindex, so every page built on it doesThe template's SEO settings

Google's own launch checklist names the classic case:

"Don't forget to remove any noindex or robots.txt blocks that were only needed for the migration." (Google Search Central: Site moves with URL changes)

For the full post-launch traffic check, see why traffic dropped after a redesign.

Don't fix it with robots.txt

robots.txt is a file that tells crawlers which pages not to visit. It is not a noindex, and it can hide one. Google warns that if a page is blocked by robots.txt, "the crawler will never see the noindex rule, and the page can still appear in search results". So:

  • To keep a page out of Google: use noindex and let Google crawl it.
  • To bring a page back: remove the noindex. Don't add a robots.txt block "to be safe".

The same applies to crawl budget, the number of pages Google crawls on your site. Google tells large sites not to use noindex to save it, since Google "will still request, but then drop the page". A small business site doesn't need to think about crawl budget.

I removed it. How fast does Google come back?

Pages return as Google recrawls them, important pages first.

  1. Confirm the noindex is gone: rerun the crawler and the live URL test.
  2. In URL Inspection, click Request indexing for your home page and top service pages.
  3. Resubmit your sitemap in Search Console so Google sees every page you want found.

Rankings come back as pages are re-indexed. Nothing guarantees a date, and less-visited pages can take longer.

Example

Illustrative cases with invented numbers.

A plumber after a redesign (US dollars)

The designer built the new site on a test address with WordPress's "Discourage search engines" box checked, then moved it live. Calls from search fell from about 40 a month to 10 over six weeks. The crawler showed noindex on all 60 pages. Unchecking one box and requesting indexing fixed the cause.

A Toronto accountant who switched SEO plugins

Search Console listed all 25 blog posts under "Excluded by 'noindex' tag", but the service pages were fine. The new plugin had the Posts type set to hidden. One toggle fixed it.

A clinic with a clean page source

The page code had no noindex, but the live URL test said "Indexing allowed? No". The response headers showed `X-Robots-Tag: noindex`, added by the host for a test copy. The host removed the rule.

FAQ

Is it bad to have some noindex pages?

No. Google lists a noindex tag among the right reasons for a page not to be indexed. Thank-you, login and checkout pages are normal.

Does a noindex checker website check my whole site?

Most free checkers test one address at a time. For every page, use Search Console's Page indexing report and a crawler.

Can a noindex page still show in Google?

Yes, until Google recrawls it, or if robots.txt blocks Google from seeing the tag.

Different for you?

Keep learning

NextSitemap priority doesn't work: get key pages crawled firstGoogle ignores the sitemap priority tag and Bing largely disregards it. What does steer crawling: accurate lastmod dates, internal links and a clean sitemap.

Do it yourself

Crawlability checklist: make every page reachable by GoogleA crawlability checklist for a small business site: find orphan pages, robots.txt blocks, broken links, redirect chains and server errors, with free tools.

Ask us one question

One line is enough. We reply by email.

Sources

  1. Google Search Central: Block Search indexing with noindex
  2. Search Console Help: Page indexing report
  3. Google Search Central: Site moves with URL changes
  4. Google Search Central: site: search operator
  5. Google Search Central: Managing crawl budget for large sites
  6. WordPress Documentation: Settings Reading Screen
  7. Yoast: How to noindex URLs
  8. Wix Help Center: Hiding your site from search results
  9. Shopify Help Center: Password page
  10. Screaming Frog: SEO Spider