Products SEO & GEO Agent Paid Ads Agent Website Agent Conversion Agent
How it works Pricing FAQ Blog Book a call

Robots.txt vs noindex: which one to use for each page

Updated · 8 min read

Short answer Use noindex to keep a page out of Google's results. Use robots.txt only to stop Google crawling pages it doesn't need to read, like endless filtered or sorted lists. Never both on one page: a robots.txt block hides the noindex, and the bare address can still show up. For private pages, use a password.

Want help applying this to your business? Talk it through on a free 30-minute call.

Book a call

Robots.txt or noindex: which one do I need?

Pick by what you want to happen. Robots.txt controls whether Google's crawler (the robot that reads pages) visits a page. Noindex controls whether a page appears in search results. Google is blunt that robots.txt is the wrong tool for hiding pages:

"This is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google." (Google Search Central: Introduction to robots.txt)
What you wantUseWhy
A page stays off Google's resultsnoindexGoogle drops the page once it reads the tag
Google stops wasting visits on near-copies (sorted or filtered lists)robots.txt DisallowGoogle never requests those addresses
Nobody but clients or staff can see itA password (login)Robots.txt and noindex are requests; a login is a lock
A duplicate version points to the main oneA canonical tagGoogle prefers it to noindex for duplicates
A page gone from results within a daySearch Console Removals tool, then noindex or deleteRemovals last about 6 months
Images or videos kept out of Googlerobots.txt DisallowGoogle only indexes media it may crawl

What's the difference between robots.txt and noindex?

Robots.txt is one plain text file at the root of your site (yourdomain.com/robots.txt). Its rules tell crawlers which addresses not to request. Noindex is a rule placed on a single page, either as a meta tag in the page's code or as an HTTP header the server sends with the file.

The difference that trips people up: robots.txt doesn't keep a page out of Google. Google can't read it, but it can learn the address from links elsewhere.

"If other pages point to your page with descriptive text, Google could still index the URL without visiting the page." (Google Search Central: Introduction to robots.txt)

That result shows your address with no description. Noindex removes the page entirely, even if other sites link to it, as long as Google can read it:

"When Googlebot crawls that page and extracts the tag or header, Google will drop that page entirely from Google Search results, regardless of whether other sites link to it." (Google Search Central: Block Search indexing with noindex)

Neither one is security. Google says obeying robots.txt is up to each crawler, and noindex only affects search engines that support it. Anyone with the link can still open the page.

Which pages get noindex, robots.txt or neither?

Most small business sites need very few of either. Here is the call for the pages owners usually ask about.

PageUseNote
Thank-you or booking confirmation pagenoindexSearchers landing there mid-flow is useless, and it can inflate your lead counts
Ad landing page that copies a service pagenoindex, or leave itAds still run; Google's ad crawler (AdsBot) checks pages separately
Your site's own search results (/?s=plumber)robots.txt DisallowEndless near-copies; Google's crawl budget guide names sorted versions as an example
Shop or listing filters and sort ordersrobots.txt DisallowSame reason; only worth it if you have many of them
Admin or login pageLeave it, or noindexAlready behind a password; a Disallow line advertises the path
Client portal, invoices, intake filesPasswordNeither tool keeps people out
Staging or test copy of the sitePassword, or noindex on every pageA Disallow alone can leave its addresses in results
Two versions of one service pageCanonical tag pointing to the main pageGoogle doesn't recommend noindex for this
Old page for a service you droppedDelete it (404) or redirect itBetter than keeping a hidden page around
PDF price sheet you don't want in resultsnoindex as an HTTP headerA PDF has no page code for a meta tag
Tag, author or date archive pagesnoindex if they're thinOptional; many small sites leave them
A page that ranks and brings leadsNeitherCheck you didn't block it by accident

On duplicates, Google's canonical guide says it doesn't recommend noindex to pick the main version "because it will completely block the page from Search". On the ad page: Google lists AdsBot as a special crawler that ignores robots.txt rules written for all crawlers (User-agent: *), so a site-wide Disallow doesn't stop it. If you write a rule naming AdsBot, Google says that rule affects how Google Ads checks your page's quality.

Why you shouldn't use both on the same page

Because the robots.txt block wins, and the noindex is never seen. Google reads a page's rules only when it crawls the page:

"If a page is disallowed from crawling through the robots.txt file, then any information about indexing or serving rules will not be found and will therefore be ignored." (Google Search Central: Robots meta tag specifications)

To take a page that's already in Google out of results:

  1. Make sure robots.txt doesn't block it.
  2. Add noindex to the page.
  3. Ask Google to recrawl it with the URL Inspection tool in Search Console. Otherwise Google says a recrawl can take months.
  4. Leave it crawlable. Once the page is gone from results, keep the noindex and don't add a Disallow; a later block would hide the noindex.

Need it gone faster? Google's Removals tool hides a result within a day, but the request lasts about 6 months. Keep the noindex, a password, or delete the page so it stays gone.

How do I add noindex or a robots.txt rule?

Start with your website builder's settings. Google itself points Wix, WordPress and Blogger users to the platform's search settings rather than editing code. Search there for "hide page from search engines": the switch adds noindex for you.

If you edit code, noindex is one line in the page's head section: a meta tag with name="robots" and content="noindex". For files with no page code, such as PDFs, the server sends the header X-Robots-Tag: noindex instead. Google says the two methods have the same effect. Writing "noindex" inside robots.txt does nothing; Google doesn't support it.

A robots.txt rule takes two lines. To keep all crawlers out of your site search results:

LineMeaning
User-agent: *The next rule is for every crawler
Disallow: /search/Don't request any address starting with /search/

Rules are case-sensitive, and the file must sit at the root of your domain. Check your work in Search Console: the robots.txt report shows the file Google fetched, and URL Inspection shows whether a page is blocked or carries noindex.

WordPress's own history shows the shift. Its "Discourage search engines from indexing this site" box used to add Disallow: / (block everything) to robots.txt; since version 5.3 it adds a noindex, nofollow tag instead (WordPress Documentation: Settings Reading Screen). If that box is ticked on a live site, untick it.

Does a small business site need to worry about crawl budget?

Almost never. Crawl budget is how much crawling Google gives one site. Google's guide is for sites with about 1 million or more pages that change weekly, or 10,000 or more that change daily. For smaller sites:

"For Google Search specifically, keeping your sitemap up to date and checking the Page Indexing report regularly is adequate." (Google Crawling Infrastructure: Optimize your crawl budget)

For large sites, the guide reverses the usual advice: to save crawling, block near-copies in robots.txt, and don't rely on noindex, because Google still has to request a page to see its noindex. That's why filters and sort orders sit in the robots.txt row above.

Example

Illustrative cases with invented numbers.

A dentist's confirmation page (US dollars)

A practice pays $40 per click on Google Ads. Its "/booking-confirmed" page shows up in Google, and some visitors land there directly, so the analytics count bookings that never happened. The page gets noindex, robots.txt is left alone, and after a recrawl the page drops out of results.

A landscaper's staging site (Canadian dollars)

A web designer builds a C$6,000 redesign on staging.example.ca and blocks it with Disallow: / in robots.txt. Weeks later, the staging home page appears in Google with no description, because a supplier linked to it. The fix: a password on staging. When the new site goes live, the designer copies the live robots.txt, not the staging one, so the real site isn't blocked.

FAQ

Will a robots.txt block remove a page that's already in Google?

No. Google says a blocked address can stay in results, shown without a description. Use noindex and keep the page crawlable until it drops out.

Does noindex stop Google from following the links on the page?

Not on its own. Google says that without a nofollow rule it may use the page's links to discover other pages.

Does robots.txt hide my admin or client pages?

No. It's a public file anyone can read, and crawlers obey it by choice. Use a login for anything private.

How long does noindex take to work?

Until Google crawls the page again. Google says that can take months for less important pages. You can ask for a recrawl in URL Inspection.

Does noindex hurt the rest of my site?

No. It only affects the page that carries it. Google even notes that indexing low-quality pages may hurt how a site ranks.

Different for you?

Keep learning

NextXML vs HTML Sitemaps: What Each Does for SEOXML vs HTML sitemaps: who reads each one, what Google and Bing say about them, and whether a small business website needs one, the other, or both.

Do it yourself

Crawlability checklist: make every page reachable by GoogleA crawlability checklist for a small business site: find orphan pages, robots.txt blocks, broken links, redirect chains and server errors, with free tools.

Ask us one question

One line is enough. We reply by email.

Sources

  1. Google Search Central: Introduction to robots.txt
  2. Google Search Central: Block Search indexing with noindex
  3. Google Search Central: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
  4. Google Search Central: Control what you share with Google Search
  5. Google Search Central: Remove a page hosted on your site from Google
  6. Google Search Central: How to specify a canonical URL with rel="canonical" and other methods
  7. Google Crawling Infrastructure: Optimize your crawl budget
  8. Google Crawling Infrastructure: How to write and submit a robots.txt file
  9. Google Crawling Infrastructure: Google's special-case crawlers
  10. WordPress Documentation: Settings Reading Screen