Products SEO & GEO Agent Paid Ads Agent Website Agent Conversion Agent
How it works Pricing FAQ Blog Book a call

Robots.txt Mistakes That Block Google, and How to Fix Them

Updated · 6 min read

Short answer Most robots.txt damage comes from a few mistakes: a Disallow: / left over from a staging site, blocked CSS or script folders, short rules that match more than intended, and rules Google ignores, like noindex. Open yoursite.com/robots.txt, check each line against this list, then confirm in Search Console's robots.txt report.

Want help applying this to your business? Talk it through on a free 30-minute call.

Book a call

The mistakes, at a glance

Open yoursite.com/robots.txt in a browser. Every line there is a request to Google's crawler (Googlebot, the program that fetches your pages). These are the lines that cost owners traffic, in rough order of damage.

MistakeWhat it doesFix
Disallow: / left from the buildBlocks the whole siteDelete the line
CSS or script folders blockedGoogle can't see the page as visitors doRemove the rule for those folders
A short rule like Disallow: /blogAlso blocks /blog-tips and /bloggerEnd folder rules with /
Disallow: /*?Blocks every address with a ? in itName the exact parameter
A Googlebot group missing your other rulesGoogle ignores your * groupCopy shared rules into it
Blocking a page to get it out of GoogleThe page can stay in resultsUnblock it and use noindex
Noindex or crawl-delay linesGoogle ignores bothUse a noindex tag; drop crawl-delay
Listing private folders to hide themAnyone can read the filePassword-protect them
The file returns a server errorGoogle pauses crawlingFix the server so the file loads

Is a leftover staging rule blocking my whole site?

Check for a line that reads exactly Disallow: / under User-agent: *. That single slash matches every page (Google: robots.txt specification). Developers often add it while a new site is being built, and it ships with the launch. Google's site-move guide lists it among the common mistakes:

"Don't forget to remove any noindex or robots.txt blocks that were only needed for the migration." (Google Search Central: Site moves with URL changes)

On WordPress, also check Settings, then Reading. The box "Discourage search engines from indexing this site" now adds a noindex tag to every page; before version 5.3 it wrote Disallow: / into robots.txt instead (WordPress.org: Settings Reading screen). Either way, a box left ticked after launch keeps the site out of Google. If traffic fell right after a new site went live, start with why traffic drops after a redesign.

Am I blocking CSS, JavaScript or images?

If a rule covers the folders holding your styles and scripts, Google sees a broken page. Google renders pages the way a browser does, and it won't run code it isn't allowed to fetch: "Google Search won't render JavaScript from blocked files or on blocked pages" (Google: JavaScript SEO basics). The cost is spelled out in its starter guide:

"If your site is hiding important components that make up your website (like CSS and JavaScript), Google might not be able to understand your pages, which means they might not show up in search results or rank well for the terms you're targeting." (Google Search Central: SEO Starter Guide)

On WordPress, theme files and uploaded images sit under /wp-content/, so Disallow: /wp-content/ blocks both. Blocking an image folder also keeps those images out of Google Images (Google: Introduction to robots.txt). Google says blocking a resource is fine only if the page doesn't depend on it.

Why does my rule block more than I meant?

Because each rule matches the start of the address, and only two wildcards exist: * for any characters and $ for the end (Google: robots.txt specification). Paths are case-sensitive.

RuleYou meantIt also blocks
Disallow: /blogThe blog folder/blog-tips, /blogger.html
Disallow: /*?Sorted or filtered copiesEvery URL with a ?, like /services?page=2
Disallow: /*.pdfPDF filesAny path containing .pdf, like /guide.pdf-download
Disallow: /Admin/The admin areaNothing at /admin/ (wrong case)

Safer versions: Disallow: /blog/ for a folder, Disallow: /*?sort= for one parameter, Disallow: /*.pdf$ for files that end in .pdf.

Why is Google ignoring my rules?

Three common causes:

  • A named group replaces the general one. A crawler follows only the group that names it most specifically; "Other groups are ignored" (Google: robots.txt specification). Add a User-agent: Googlebot group with one rule, and Googlebot stops reading everything under User-agent: *.
  • The rule isn't one Google reads. Google supports user-agent, allow, disallow and sitemap only; "other fields such as crawl-delay aren't supported" (Google: robots.txt specification). Google stopped honoring noindex lines in robots.txt on September 1, 2019 (Google Search Central Blog).
  • The file is in the wrong place. It must sit at the root of each host. A file in a subfolder is never read, and blog.example.com needs its own (Google: robots.txt specification).

I blocked a page, so why is it still in Google?

Because robots.txt stops crawling, not indexing. Google can list a blocked page from links elsewhere, usually without a description. Search Console's Page indexing report says what to do:

"If you do want to block this page from Google Search, robots.txt is not the correct mechanism to avoid being indexed. To avoid being indexed, remove the robots.txt block and use 'noindex'." (Search Console Help: Page indexing report)

Blocking and noindexing the same page fails too: "the crawler will never see the noindex rule" (Google: Block indexing). For pages you need gone fast, see removing information from Google.

What if the file itself errors?

How the server answers matters as much as what the file says (Google: robots.txt specification):

Server answer for /robots.txtWhat Google does
404 or other 4xx (except 429)Crawls as if there are no rules
5xx server errorStops crawling the site for 12 hours, then uses the last good copy for up to 30 days
More than five redirectsTreats it as missing (404)

So a missing file is harmless; a file that errors can stall crawling. A firewall or security plugin that blocks Googlebot from the file can cause this.

How do I check and fix it?

  1. Read the live file at yoursite.com/robots.txt, in a private browser window.
  2. Open the robots.txt report in Search Console. Google names it as the tool for checking a live file (Google: Create a robots.txt file). Google's older robots.txt Tester was retired in late 2023, as Search Engine Roundtable reported.
  3. Inspect one key page with the URL Inspection tool to see whether crawling is allowed. See the Search Console reports worth reading.
  4. Edit and upload the corrected file. Google refreshes its copy about every 24 hours; for faster pickup, use Request a recrawl in the robots.txt report (Google: Update your robots.txt file).

Example

A roofer after a site rebuild (US dollars)

The new $6,000 site launches and calls from search stop within weeks. The robots.txt reads User-agent: * then Disallow: /. Deleting the second line and requesting a recrawl lets Google back in; pages can come back as Google recrawls them.

A bakery hiding its order page (Canadian dollars)

The owner adds Disallow: /order to keep a $40 custom-cake form out of Google. It also blocks /ordering-faq, a page that ranked. Changing it to Disallow: /order$ blocks only the form. Better still: drop the rule and put a noindex tag on the form, since that is what keeps a page out of results.

FAQ

Can one wrong line really hide my whole site?

Yes. Disallow: / under User-agent: * asks every crawler to skip every page.

Should I block /wp-admin/ on WordPress?

Blocking the admin area is common and harmless, as long as the rule doesn't also cover /wp-content/, where theme styles and images live.

Does Google obey crawl-delay?

No. Google ignores it (Google: robots.txt specification).

Can robots.txt keep private pages safe?

No. Anyone can read the file, and bad bots ignore it. Google says to password-protect private files instead (Google: Introduction to robots.txt).

How fast does Google see my fix?

Usually within a day, since Google caches the file for up to 24 hours. A recrawl request in the robots.txt report can speed it up.

Different for you?

Keep learning

NextCrawlability checklist: make every page reachable by GoogleA crawlability checklist for a small business site: find orphan pages, robots.txt blocks, broken links, redirect chains and server errors, with free tools.

Do it yourself

Core Web Vitals Checklist: Quick Wins for a Faster SiteA one-week Core Web Vitals checklist for a small business site: find the failing metric, then the image, layout and script fixes an owner can make.

Ask us one question

One line is enough. We reply by email.

Sources

  1. Google Crawling Infrastructure: How Google interprets the robots.txt specification
  2. Google Search Central: Introduction to robots.txt
  3. Google Search Central: Site moves with URL changes
  4. Google Search Central: SEO Starter Guide
  5. Google Search Central: Understand the JavaScript SEO basics
  6. Search Console Help: Page indexing report
  7. Google Search Central: Block Search indexing with noindex
  8. Google Search Central Blog: A note on unsupported rules in robots.txt (July 2019)
  9. Google Search Central: How to write and submit a robots.txt file
  10. Google Search Central: Update your robots.txt file
  11. WordPress.org Documentation: Settings Reading screen