Products SEO & GEO Agent Paid Ads Agent Website Agent Conversion Agent
How it works Pricing FAQ Blog Book a call

Noindex checker: test any URL for a noindex tag

Updated · 6 min read

Short answer A noindex rule can hide in two places: a robots meta tag in the page's code, or an X-Robots-Tag header the server sends. Check both. View the page source and search for "noindex", then read the response headers in your browser or with curl. For pages on your own site, Google's URL Inspection shows exactly what Googlebot saw.

Want help applying this to your business? Talk it through on a free 30-minute call.

Book a call

How do I check if a page has noindex?

Look in two places, because Google reads both. Google's Search Central documentation names two ways to set noindex.

One is a meta tag in the page's code, the other an HTTP response header:

"They have the same effect; choose the method that is more convenient for your site and appropriate for the content type." (Google Search Central: Block Search indexing with noindex)

A checker that reads only the page's code misses the header, so check both.

Where it hidesWhat it looks likeHow to see it
Robots meta tag, in the page's HTML code`meta name="robots" content="noindex"`View page source, search for "noindex"
Google-only meta tag`meta name="googlebot" content="noindex"`Same search
X-Robots-Tag HTTP header (a note the server sends with the file, invisible on the page)`X-Robots-Tag: noindex`Browser developer tools or curl
"none" in either place`content="none"`Google treats it as "noindex, nofollow"

Google's robots meta tag specification says the words are not case sensitive, so "NOINDEX" counts too. It also says Google respects a robots meta tag placed in the page body, not just the head, so search the whole source.

Check one URL by hand in your browser

  1. Open the page. Right-click and choose View Page Source (or type `view-source:` before the address in Chrome).
  2. Press Ctrl+F (Cmd+F on a Mac) and search for noindex, then for none. Any match inside a meta tag named `robots` or `googlebot` means the page asks not to be indexed.
  3. For the header, open Chrome's developer tools (F12), go to the Network tab and reload. Click the first row (the page itself), open Headers and scroll to Response Headers. Chrome's DevTools guide describes this exact path. Look for a line named x-robots-tag.

Check PDFs this way too. A PDF has no HTML code, so the header is the only place a noindex can sit. Google says the header "can be used for non-HTML resources, such as PDFs, video files, and image files" (Google Search Central: Block Search indexing with noindex).

Check with one command

If you're comfortable with a terminal, curl answers both questions. The curl manual describes `-I` as: "Fetch the headers only."

  • Header: `curl -sIL https://yourdomain.com/page/ | grep -i x-robots-tag`
  • Meta tag: `curl -sL https://yourdomain.com/page/ | grep -i noindex`

The `-L` follows redirects to the final page. No output from either means no noindex in the code the server sent. A match from the second one is worth a look: the word may sit in page text, not a tag. To check a list, save one address per line in urls.txt and run: `cat urls.txt | while read u; do echo "$u"; curl -sIL "$u" | grep -i x-robots-tag; curl -sL "$u" | grep -i noindex; done`

Check what Google actually saw

For pages on a site you own, Search Console is the final word. It's free: "Google Search Console is a free service offered by Google" (Search Console Help: About Search Console). Google's noindex page recommends two of its tools:

"To test if your noindex implementation is correct, use the URL Inspection tool to see the HTML that Googlebot received while crawling the page. You can also use the Page Indexing report in Search Console to monitor the pages on your site from which Googlebot extracted a noindex rule." (Google Search Central: Block Search indexing with noindex)

In URL Inspection, read the line Indexing allowed? A "No" on a page you want found means a noindex is blocking it (Search Console Help: URL Inspection tool). For a whole-site list, open the Page indexing report and look for the reason about a noindex tag. What each reason means: Crawled or discovered, currently not indexed.

What do free online noindex checkers miss?

A quick checker is fine for a spot check. Know its blind spots:

Blind spotWhy it matters
Header not checkedSome read only the HTML; a server or plugin rule in the header slips past
Tags added by JavaScriptGoogle's JavaScript guide shows code that adds a robots meta tag after the page loads. A tool that reads only the raw code misses it
JavaScript that removes noindexGoogle says it "may skip rendering and JavaScript execution" when it meets noindex, so removing the tag with code "may not work as expected" (Google Search Central: JavaScript SEO basics). The noindex must be gone from the original code
Robots.txt blocking the pageGoogle can't read a noindex on a page it may not crawl, so the rule "will therefore be ignored" (Google Search Central: Robots meta tag specifications). A checker shows the tag; Google never sees it
Other search enginesGoogle warns "some search engines might interpret the noindex rule differently" (Google Search Central: Block Search indexing with noindex)

The robots.txt case has a twist in URL Inspection: when robots.txt blocks a page, "Indexing allowed?" always reads Yes, because Google can't see the tag (Search Console Help: URL Inspection tool).

Found a noindex you didn't want?

  1. Find where it comes from. On WordPress, check Settings, then Reading. The box "Discourage search engines from indexing this site" adds a noindex, nofollow tag to every page (WordPress Documentation: Settings Reading Screen). It's often left on after a redesign. Otherwise look at your SEO plugin's per-page setting, your theme, or your host's header rules.
  2. Remove it from the original code, not with a script that runs later (see the JavaScript row above).
  3. Make sure robots.txt doesn't block the page. Google has to crawl it to notice the change.
  4. Ask Google to recrawl with URL Inspection. Google says that without it, "it may take months for Googlebot to revisit a page" (Google Search Central: Block Search indexing with noindex).

Adding noindex to save Google's crawling time doesn't work either. Google says "Don't use noindex", because it still requests the page and then drops it, "wasting crawling time" (Google Crawling Infrastructure: Optimize your crawl budget). That advice is for very large sites anyway.

Example

Illustrative cases.

A roofer's new WordPress site

The service pages never appear on Google two months after launch. View Page Source on the home page shows `content='noindex, nofollow'`. The developer left "Discourage search engines" checked. Unchecking it, then requesting indexing for the five main pages in URL Inspection, is the whole fix.

An accountant's price list PDF

The PDF ranks, but the owner wants it out because prices changed. Its HTML can't carry a meta tag, so the host adds `X-Robots-Tag: noindex` to that file. `curl -sI` on the PDF's address confirms the header is sent.

A clinic's booking page built with JavaScript

An online checker says "indexable", but URL Inspection says indexing isn't allowed. The page's starting code carries noindex, and a script removes it after loading. Google acted on the original code. Removing the tag from the template fixes it.

FAQ

Is a noindex checker the same as checking if a page is indexed?

No. A noindex checker tells you whether the page asks not to be indexed. A page without noindex can still be left out of Google for other reasons. URL Inspection shows the actual index status.

Does nofollow stop a page from being indexed?

No. Nofollow tells Google not to follow the page's links. Noindex keeps the page out of results, and so does "none", which Google treats as noindex plus nofollow (Google Search Central: Robots meta tag specifications).

Can I put noindex in robots.txt?

No. Google says "Specifying the noindex rule in the robots.txt file is not supported by Google" (Google Search Central: Block Search indexing with noindex).

How long after removing noindex does a page come back?

Only after Google crawls it again. Request a recrawl in URL Inspection; without one, Google says a revisit can take months.

Different for you?

Keep learning

NextSitemap finder: how to find any website's sitemapFind any website's sitemap in under a minute: check robots.txt, try the common file names, then use the free and paid tools, and know what each one misses.

Do it yourself

Crawlability checklist: make every page reachable by GoogleA crawlability checklist for a small business site: find orphan pages, robots.txt blocks, broken links, redirect chains and server errors, with free tools.

Ask us one question

One line is enough. We reply by email.

Sources

  1. Google Search Central: Block Search indexing with noindex
  2. Google Search Central: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
  3. Google Search Central: Understand the JavaScript SEO basics
  4. Google Crawling Infrastructure: Optimize your crawl budget
  5. Search Console Help: URL Inspection tool
  6. Search Console Help: About Search Console
  7. Chrome DevTools: Network features reference
  8. curl man page
  9. WordPress Documentation: Settings Reading Screen