Robots.txt Mistakes That Block Google, and How to Fix Them
Short answer Most robots.txt damage comes from a few mistakes: a Disallow: / left over from a staging site, blocked CSS or script folders, short rules that match more than intended, and rules Google ignores, like noindex. Open yoursite.com/robots.txt, check each line against this list, then confirm in Search Console's robots.txt report.
Want help applying this to your business? Talk it through on a free 30-minute call.
Book a callThe mistakes, at a glance
Open yoursite.com/robots.txt in a browser. Every line there is a request to Google's crawler (Googlebot, the program that fetches your pages). These are the lines that cost owners traffic, in rough order of damage.
| Mistake | What it does | Fix |
|---|---|---|
| Disallow: / left from the build | Blocks the whole site | Delete the line |
| CSS or script folders blocked | Google can't see the page as visitors do | Remove the rule for those folders |
| A short rule like Disallow: /blog | Also blocks /blog-tips and /blogger | End folder rules with / |
| Disallow: /*? | Blocks every address with a ? in it | Name the exact parameter |
| A Googlebot group missing your other rules | Google ignores your * group | Copy shared rules into it |
| Blocking a page to get it out of Google | The page can stay in results | Unblock it and use noindex |
| Noindex or crawl-delay lines | Google ignores both | Use a noindex tag; drop crawl-delay |
| Listing private folders to hide them | Anyone can read the file | Password-protect them |
| The file returns a server error | Google pauses crawling | Fix the server so the file loads |
Is a leftover staging rule blocking my whole site?
Check for a line that reads exactly Disallow: / under User-agent: *. That single slash matches every page (Google: robots.txt specification). Developers often add it while a new site is being built, and it ships with the launch. Google's site-move guide lists it among the common mistakes:
"Don't forget to remove any noindex or robots.txt blocks that were only needed for the migration." (Google Search Central: Site moves with URL changes)
On WordPress, also check Settings, then Reading. The box "Discourage search engines from indexing this site" now adds a noindex tag to every page; before version 5.3 it wrote Disallow: / into robots.txt instead (WordPress.org: Settings Reading screen). Either way, a box left ticked after launch keeps the site out of Google. If traffic fell right after a new site went live, start with why traffic drops after a redesign.
Am I blocking CSS, JavaScript or images?
If a rule covers the folders holding your styles and scripts, Google sees a broken page. Google renders pages the way a browser does, and it won't run code it isn't allowed to fetch: "Google Search won't render JavaScript from blocked files or on blocked pages" (Google: JavaScript SEO basics). The cost is spelled out in its starter guide:
"If your site is hiding important components that make up your website (like CSS and JavaScript), Google might not be able to understand your pages, which means they might not show up in search results or rank well for the terms you're targeting." (Google Search Central: SEO Starter Guide)
On WordPress, theme files and uploaded images sit under /wp-content/, so Disallow: /wp-content/ blocks both. Blocking an image folder also keeps those images out of Google Images (Google: Introduction to robots.txt). Google says blocking a resource is fine only if the page doesn't depend on it.
Why does my rule block more than I meant?
Because each rule matches the start of the address, and only two wildcards exist: * for any characters and $ for the end (Google: robots.txt specification). Paths are case-sensitive.
| Rule | You meant | It also blocks |
|---|---|---|
| Disallow: /blog | The blog folder | /blog-tips, /blogger.html |
| Disallow: /*? | Sorted or filtered copies | Every URL with a ?, like /services?page=2 |
| Disallow: /*.pdf | PDF files | Any path containing .pdf, like /guide.pdf-download |
| Disallow: /Admin/ | The admin area | Nothing at /admin/ (wrong case) |
Safer versions: Disallow: /blog/ for a folder, Disallow: /*?sort= for one parameter, Disallow: /*.pdf$ for files that end in .pdf.
Why is Google ignoring my rules?
Three common causes:
- A named group replaces the general one. A crawler follows only the group that names it most specifically; "Other groups are ignored" (Google: robots.txt specification). Add a User-agent: Googlebot group with one rule, and Googlebot stops reading everything under User-agent: *.
- The rule isn't one Google reads. Google supports user-agent, allow, disallow and sitemap only; "other fields such as crawl-delay aren't supported" (Google: robots.txt specification). Google stopped honoring noindex lines in robots.txt on September 1, 2019 (Google Search Central Blog).
- The file is in the wrong place. It must sit at the root of each host. A file in a subfolder is never read, and blog.example.com needs its own (Google: robots.txt specification).
I blocked a page, so why is it still in Google?
Because robots.txt stops crawling, not indexing. Google can list a blocked page from links elsewhere, usually without a description. Search Console's Page indexing report says what to do:
"If you do want to block this page from Google Search, robots.txt is not the correct mechanism to avoid being indexed. To avoid being indexed, remove the robots.txt block and use 'noindex'." (Search Console Help: Page indexing report)
Blocking and noindexing the same page fails too: "the crawler will never see the noindex rule" (Google: Block indexing). For pages you need gone fast, see removing information from Google.
What if the file itself errors?
How the server answers matters as much as what the file says (Google: robots.txt specification):
| Server answer for /robots.txt | What Google does |
|---|---|
| 404 or other 4xx (except 429) | Crawls as if there are no rules |
| 5xx server error | Stops crawling the site for 12 hours, then uses the last good copy for up to 30 days |
| More than five redirects | Treats it as missing (404) |
So a missing file is harmless; a file that errors can stall crawling. A firewall or security plugin that blocks Googlebot from the file can cause this.
How do I check and fix it?
- Read the live file at yoursite.com/robots.txt, in a private browser window.
- Open the robots.txt report in Search Console. Google names it as the tool for checking a live file (Google: Create a robots.txt file). Google's older robots.txt Tester was retired in late 2023, as Search Engine Roundtable reported.
- Inspect one key page with the URL Inspection tool to see whether crawling is allowed. See the Search Console reports worth reading.
- Edit and upload the corrected file. Google refreshes its copy about every 24 hours; for faster pickup, use Request a recrawl in the robots.txt report (Google: Update your robots.txt file).
Example
A roofer after a site rebuild (US dollars)
The new $6,000 site launches and calls from search stop within weeks. The robots.txt reads User-agent: * then Disallow: /. Deleting the second line and requesting a recrawl lets Google back in; pages can come back as Google recrawls them.
A bakery hiding its order page (Canadian dollars)
The owner adds Disallow: /order to keep a $40 custom-cake form out of Google. It also blocks /ordering-faq, a page that ranked. Changing it to Disallow: /order$ blocks only the form. Better still: drop the rule and put a noindex tag on the form, since that is what keeps a page out of results.
FAQ
Can one wrong line really hide my whole site?
Yes. Disallow: / under User-agent: * asks every crawler to skip every page.
Should I block /wp-admin/ on WordPress?
Blocking the admin area is common and harmless, as long as the rule doesn't also cover /wp-content/, where theme styles and images live.
Does Google obey crawl-delay?
No. Google ignores it (Google: robots.txt specification).
Can robots.txt keep private pages safe?
No. Anyone can read the file, and bad bots ignore it. Google says to password-protect private files instead (Google: Introduction to robots.txt).
How fast does Google see my fix?
Usually within a day, since Google caches the file for up to 24 hours. A recrawl request in the robots.txt report can speed it up.
Different for you?
- Your site is missing from Google and robots.txt looks fine: work through why a website isn't showing on Google.
- Traffic fell after a new site or new host: see traffic drops after a redesign.
- You want to block AI bots without hurting Google: see blocking AI crawlers.
- You'd rather someone else keep crawl settings right: see /seo/.