Products SEO & GEO Agent Paid Ads Agent Website Agent Conversion Agent
How it works Pricing FAQ Blog Book a call

Robots.txt Plus Noindex: The Mistake That Keeps Pages in Google

Updated · 4 min read

Short answer Don't use both on the same page. Google has to crawl a page to read its noindex tag, and a robots.txt block stops the crawl, so Google never sees the tag. The URL can stay in search, often with no description. To remove a page: delete the robots.txt rule, keep noindex, and let Google recrawl.

Want help applying this to your business? Talk it through on a free 30-minute call.

Book a call

Why robots.txt plus noindex keeps a page in Google

The two rules cancel each other. Robots.txt is a file at the root of your site that tells search engine robots which addresses they may visit. Noindex is a line inside one page (or in the server's reply for that page) that says "don't list me in search". To read the noindex, Google must visit the page. Block the visit, and the noindex is never read. Google says it plainly:

"If the page is blocked by a robots.txt file or the crawler can't access the page, the crawler will never see the noindex rule, and the page can still appear in search results, for example if other pages link to it." (Google Search Central: Block Search indexing with noindex)

The same rule holds for every indexing instruction, not just noindex: "If indexing or serving rules must be followed, the URLs containing those rules cannot be disallowed from crawling" (Google Search Central: Robots meta tag specifications).

Doesn't blocking a page in robots.txt remove it from Google?

No. Robots.txt controls visits, not listings. Google calls it "not a mechanism for keeping a web page out of Google" (Google Search Central: Introduction to robots.txt). If other pages link to a blocked address, Google can list it without ever opening it. What you'd see in search: the bare address with no description under it (same page).

Which one should I use?

GoalUseWhy
Keep a page out of search resultsNoindex onlyGoogle must crawl it to obey it
Keep private content away from everyoneA password or loginGoogle lists password-protection as a removal method (Google Search Central)
Stop robots visiting pages that waste their time or load your serverRobots.txt onlyThat's its main job (Google Search Central)
Keep images, video or audio out of GoogleRobots.txtGoogle says this works for media files (same page)
Remove a page todayThe Removals tool in Search Console, then noindexThe tool hides it "within a day"; noindex makes it stick (Google Search Central)

One more trap: putting `noindex` inside robots.txt. "Specifying the noindex rule in the robots.txt file is not supported by Google" (Google Search Central).

How to fix it, in order

  1. Delete the robots.txt rule that blocks the page. On Wix, Blogger and similar builders you may not edit the file directly; look for a search settings page instead (Google Search Central).
  2. Keep the noindex on the page: a robots meta tag set to `noindex` in the page's head section, or the `X-Robots-Tag: noindex` header. Google says both "have the same effect" (Google Search Central).
  3. Check what Google sees. In Search Console, run URL Inspection on the page; Google suggests it to see "the HTML that Googlebot received" (same page).
  4. Request a recrawl in URL Inspection. Without it, Google says a revisit "may take months" for a page few sites link to (same page).
  5. Wait for the drop. Once Google reads the tag, it removes the page "regardless of whether other sites link to it" (same page).
  6. Add the robots.txt block back only if you must, after the page has left search. Google ignores the noindex from then on, so the page can return if new links point to it.

What Search Console shows

The Page indexing report names this exact problem. Indexed, though blocked by robots.txt means Google listed the page without visiting it. Its advice:

"To avoid being indexed, remove the robots.txt block and use 'noindex'." (Search Console Help: Page indexing report)

URL marked 'noindex' is the status you want for pages you hid on purpose. For what every other status means, see our guide to Page indexing statuses.

Example

A thank-you page that won't leave

A cleaning company adds noindex to its "thanks for booking" page and also blocks it in robots.txt to be safe. Months later it still appears in Google with no description. They delete the robots.txt line, request a recrawl, and the page drops out after Google revisits.

An old price list

A clinic's 2024 price list sits blocked in robots.txt. Search Console shows "Indexed, though blocked by robots.txt". Removing the block and adding noindex is the fix; the Removals tool hides it within a day while they wait.

FAQ

Can Google index a page it never crawled?

Yes. If other pages link to the address, Google can list it from those links alone, usually with little or no description (Search Console Help).

Should I add nofollow too?

Only if Google should also ignore the page's links. `noindex, nofollow` is allowed together (Google Search Central). It doesn't change whether the page is indexed.

How long until a noindexed page disappears?

After Google next crawls it. That can take months for a little-linked page, so request a recrawl in URL Inspection (Google Search Central).

Do other search engines obey noindex?

Google notes that search engines "that don't support noindex" can still show the page (Google Search Central). For anything truly private, use a password.

Different for you?

Keep learning

NextRobots.txt: What It Is, How It Works, and What It Can't DoWhat a robots.txt file is, how its lines work, which rule wins, and the jobs it can't do: hide pages from Google, protect private files or stop bad bots.

Do it yourself

Crawlability checklist: make every page reachable by GoogleA crawlability checklist for a small business site: find orphan pages, robots.txt blocks, broken links, redirect chains and server errors, with free tools.

Ask us one question

One line is enough. We reply by email.

Sources

  1. Google Search Central: Block Search indexing with noindex
  2. Google Search Central: Introduction to robots.txt
  3. Google Search Central: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
  4. Google Search Central: Remove a page hosted on your site from Google
  5. Search Console Help: Page indexing report