Robots.txt vs noindex: which one to use for each page
Short answer Use noindex to keep a page out of Google's results. Use robots.txt only to stop Google crawling pages it doesn't need to read, like endless filtered or sorted lists. Never both on one page: a robots.txt block hides the noindex, and the bare address can still show up. For private pages, use a password.
Want help applying this to your business? Talk it through on a free 30-minute call.
Book a callRobots.txt or noindex: which one do I need?
Pick by what you want to happen. Robots.txt controls whether Google's crawler (the robot that reads pages) visits a page. Noindex controls whether a page appears in search results. Google is blunt that robots.txt is the wrong tool for hiding pages:
"This is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google." (Google Search Central: Introduction to robots.txt)
| What you want | Use | Why |
|---|---|---|
| A page stays off Google's results | noindex | Google drops the page once it reads the tag |
| Google stops wasting visits on near-copies (sorted or filtered lists) | robots.txt Disallow | Google never requests those addresses |
| Nobody but clients or staff can see it | A password (login) | Robots.txt and noindex are requests; a login is a lock |
| A duplicate version points to the main one | A canonical tag | Google prefers it to noindex for duplicates |
| A page gone from results within a day | Search Console Removals tool, then noindex or delete | Removals last about 6 months |
| Images or videos kept out of Google | robots.txt Disallow | Google only indexes media it may crawl |
What's the difference between robots.txt and noindex?
Robots.txt is one plain text file at the root of your site (yourdomain.com/robots.txt). Its rules tell crawlers which addresses not to request. Noindex is a rule placed on a single page, either as a meta tag in the page's code or as an HTTP header the server sends with the file.
The difference that trips people up: robots.txt doesn't keep a page out of Google. Google can't read it, but it can learn the address from links elsewhere.
"If other pages point to your page with descriptive text, Google could still index the URL without visiting the page." (Google Search Central: Introduction to robots.txt)
That result shows your address with no description. Noindex removes the page entirely, even if other sites link to it, as long as Google can read it:
"When Googlebot crawls that page and extracts the tag or header, Google will drop that page entirely from Google Search results, regardless of whether other sites link to it." (Google Search Central: Block Search indexing with noindex)
Neither one is security. Google says obeying robots.txt is up to each crawler, and noindex only affects search engines that support it. Anyone with the link can still open the page.
Which pages get noindex, robots.txt or neither?
Most small business sites need very few of either. Here is the call for the pages owners usually ask about.
| Page | Use | Note |
|---|---|---|
| Thank-you or booking confirmation page | noindex | Searchers landing there mid-flow is useless, and it can inflate your lead counts |
| Ad landing page that copies a service page | noindex, or leave it | Ads still run; Google's ad crawler (AdsBot) checks pages separately |
| Your site's own search results (/?s=plumber) | robots.txt Disallow | Endless near-copies; Google's crawl budget guide names sorted versions as an example |
| Shop or listing filters and sort orders | robots.txt Disallow | Same reason; only worth it if you have many of them |
| Admin or login page | Leave it, or noindex | Already behind a password; a Disallow line advertises the path |
| Client portal, invoices, intake files | Password | Neither tool keeps people out |
| Staging or test copy of the site | Password, or noindex on every page | A Disallow alone can leave its addresses in results |
| Two versions of one service page | Canonical tag pointing to the main page | Google doesn't recommend noindex for this |
| Old page for a service you dropped | Delete it (404) or redirect it | Better than keeping a hidden page around |
| PDF price sheet you don't want in results | noindex as an HTTP header | A PDF has no page code for a meta tag |
| Tag, author or date archive pages | noindex if they're thin | Optional; many small sites leave them |
| A page that ranks and brings leads | Neither | Check you didn't block it by accident |
On duplicates, Google's canonical guide says it doesn't recommend noindex to pick the main version "because it will completely block the page from Search". On the ad page: Google lists AdsBot as a special crawler that ignores robots.txt rules written for all crawlers (User-agent: *), so a site-wide Disallow doesn't stop it. If you write a rule naming AdsBot, Google says that rule affects how Google Ads checks your page's quality.
Why you shouldn't use both on the same page
Because the robots.txt block wins, and the noindex is never seen. Google reads a page's rules only when it crawls the page:
"If a page is disallowed from crawling through the robots.txt file, then any information about indexing or serving rules will not be found and will therefore be ignored." (Google Search Central: Robots meta tag specifications)
To take a page that's already in Google out of results:
- Make sure robots.txt doesn't block it.
- Add noindex to the page.
- Ask Google to recrawl it with the URL Inspection tool in Search Console. Otherwise Google says a recrawl can take months.
- Leave it crawlable. Once the page is gone from results, keep the noindex and don't add a Disallow; a later block would hide the noindex.
Need it gone faster? Google's Removals tool hides a result within a day, but the request lasts about 6 months. Keep the noindex, a password, or delete the page so it stays gone.
How do I add noindex or a robots.txt rule?
Start with your website builder's settings. Google itself points Wix, WordPress and Blogger users to the platform's search settings rather than editing code. Search there for "hide page from search engines": the switch adds noindex for you.
If you edit code, noindex is one line in the page's head section: a meta tag with name="robots" and content="noindex". For files with no page code, such as PDFs, the server sends the header X-Robots-Tag: noindex instead. Google says the two methods have the same effect. Writing "noindex" inside robots.txt does nothing; Google doesn't support it.
A robots.txt rule takes two lines. To keep all crawlers out of your site search results:
| Line | Meaning |
|---|---|
| User-agent: * | The next rule is for every crawler |
| Disallow: /search/ | Don't request any address starting with /search/ |
Rules are case-sensitive, and the file must sit at the root of your domain. Check your work in Search Console: the robots.txt report shows the file Google fetched, and URL Inspection shows whether a page is blocked or carries noindex.
WordPress's own history shows the shift. Its "Discourage search engines from indexing this site" box used to add Disallow: / (block everything) to robots.txt; since version 5.3 it adds a noindex, nofollow tag instead (WordPress Documentation: Settings Reading Screen). If that box is ticked on a live site, untick it.
Does a small business site need to worry about crawl budget?
Almost never. Crawl budget is how much crawling Google gives one site. Google's guide is for sites with about 1 million or more pages that change weekly, or 10,000 or more that change daily. For smaller sites:
"For Google Search specifically, keeping your sitemap up to date and checking the Page Indexing report regularly is adequate." (Google Crawling Infrastructure: Optimize your crawl budget)
For large sites, the guide reverses the usual advice: to save crawling, block near-copies in robots.txt, and don't rely on noindex, because Google still has to request a page to see its noindex. That's why filters and sort orders sit in the robots.txt row above.
Example
Illustrative cases with invented numbers.
A dentist's confirmation page (US dollars)
A practice pays $40 per click on Google Ads. Its "/booking-confirmed" page shows up in Google, and some visitors land there directly, so the analytics count bookings that never happened. The page gets noindex, robots.txt is left alone, and after a recrawl the page drops out of results.
A landscaper's staging site (Canadian dollars)
A web designer builds a C$6,000 redesign on staging.example.ca and blocks it with Disallow: / in robots.txt. Weeks later, the staging home page appears in Google with no description, because a supplier linked to it. The fix: a password on staging. When the new site goes live, the designer copies the live robots.txt, not the staging one, so the real site isn't blocked.
FAQ
Will a robots.txt block remove a page that's already in Google?
No. Google says a blocked address can stay in results, shown without a description. Use noindex and keep the page crawlable until it drops out.
Does noindex stop Google from following the links on the page?
Not on its own. Google says that without a nofollow rule it may use the page's links to discover other pages.
Does robots.txt hide my admin or client pages?
No. It's a public file anyone can read, and crawlers obey it by choice. Use a login for anything private.
How long does noindex take to work?
Until Google crawls the page again. Google says that can take months for less important pages. You can ask for a recrawl in URL Inspection.
Does noindex hurt the rest of my site?
No. It only affects the page that carries it. Google even notes that indexing low-quality pages may hurt how a site ranks.
Different for you?
- Search Console says a page is excluded by noindex or blocked by robots.txt and you didn't mean it: crawled or discovered, currently not indexed.
- Your whole site is missing from Google: website not showing up on Google.
- You want old or personal information off Google: remove information from Google.
- You're deciding whether to block ChatGPT or other AI bots: should you block AI crawlers?.
- You'd rather have your site's crawling and indexing checked for you: SEO & GEO Agent.