Robots.txt Plus Noindex: The Mistake That Keeps Pages in Google
Short answer Don't use both on the same page. Google has to crawl a page to read its noindex tag, and a robots.txt block stops the crawl, so Google never sees the tag. The URL can stay in search, often with no description. To remove a page: delete the robots.txt rule, keep noindex, and let Google recrawl.
Want help applying this to your business? Talk it through on a free 30-minute call.
Book a callWhy robots.txt plus noindex keeps a page in Google
The two rules cancel each other. Robots.txt is a file at the root of your site that tells search engine robots which addresses they may visit. Noindex is a line inside one page (or in the server's reply for that page) that says "don't list me in search". To read the noindex, Google must visit the page. Block the visit, and the noindex is never read. Google says it plainly:
"If the page is blocked by a robots.txt file or the crawler can't access the page, the crawler will never see the noindex rule, and the page can still appear in search results, for example if other pages link to it." (Google Search Central: Block Search indexing with noindex)
The same rule holds for every indexing instruction, not just noindex: "If indexing or serving rules must be followed, the URLs containing those rules cannot be disallowed from crawling" (Google Search Central: Robots meta tag specifications).
Doesn't blocking a page in robots.txt remove it from Google?
No. Robots.txt controls visits, not listings. Google calls it "not a mechanism for keeping a web page out of Google" (Google Search Central: Introduction to robots.txt). If other pages link to a blocked address, Google can list it without ever opening it. What you'd see in search: the bare address with no description under it (same page).
Which one should I use?
| Goal | Use | Why |
|---|---|---|
| Keep a page out of search results | Noindex only | Google must crawl it to obey it |
| Keep private content away from everyone | A password or login | Google lists password-protection as a removal method (Google Search Central) |
| Stop robots visiting pages that waste their time or load your server | Robots.txt only | That's its main job (Google Search Central) |
| Keep images, video or audio out of Google | Robots.txt | Google says this works for media files (same page) |
| Remove a page today | The Removals tool in Search Console, then noindex | The tool hides it "within a day"; noindex makes it stick (Google Search Central) |
One more trap: putting `noindex` inside robots.txt. "Specifying the noindex rule in the robots.txt file is not supported by Google" (Google Search Central).
How to fix it, in order
- Delete the robots.txt rule that blocks the page. On Wix, Blogger and similar builders you may not edit the file directly; look for a search settings page instead (Google Search Central).
- Keep the noindex on the page: a robots meta tag set to `noindex` in the page's head section, or the `X-Robots-Tag: noindex` header. Google says both "have the same effect" (Google Search Central).
- Check what Google sees. In Search Console, run URL Inspection on the page; Google suggests it to see "the HTML that Googlebot received" (same page).
- Request a recrawl in URL Inspection. Without it, Google says a revisit "may take months" for a page few sites link to (same page).
- Wait for the drop. Once Google reads the tag, it removes the page "regardless of whether other sites link to it" (same page).
- Add the robots.txt block back only if you must, after the page has left search. Google ignores the noindex from then on, so the page can return if new links point to it.
What Search Console shows
The Page indexing report names this exact problem. Indexed, though blocked by robots.txt means Google listed the page without visiting it. Its advice:
"To avoid being indexed, remove the robots.txt block and use 'noindex'." (Search Console Help: Page indexing report)
URL marked 'noindex' is the status you want for pages you hid on purpose. For what every other status means, see our guide to Page indexing statuses.
Example
A thank-you page that won't leave
A cleaning company adds noindex to its "thanks for booking" page and also blocks it in robots.txt to be safe. Months later it still appears in Google with no description. They delete the robots.txt line, request a recrawl, and the page drops out after Google revisits.
An old price list
A clinic's 2024 price list sits blocked in robots.txt. Search Console shows "Indexed, though blocked by robots.txt". Removing the block and adding noindex is the fix; the Removals tool hides it within a day while they wait.
FAQ
Can Google index a page it never crawled?
Yes. If other pages link to the address, Google can list it from those links alone, usually with little or no description (Search Console Help).
Should I add nofollow too?
Only if Google should also ignore the page's links. `noindex, nofollow` is allowed together (Google Search Central). It doesn't change whether the page is indexed.
How long until a noindexed page disappears?
After Google next crawls it. That can take months for a little-linked page, so request a recrawl in URL Inspection (Google Search Central).
Do other search engines obey noindex?
Google notes that search engines "that don't support noindex" can still show the page (Google Search Central). For anything truly private, use a password.
Different for you?
- Not sure the page is indexed at all? Start with the Search Console basics.
- Blocking AI bots, not Google Search? That's a robots.txt choice with other trade-offs: blocking AI crawlers.
- Rankings fell after a new site launched? Check for a robots.txt block left over from the test site: traffic drop after a redesign.
- Want someone to audit crawl and index settings? See /seo/.