Robots.txt: What It Is, How It Works, and What It Can't Do
Short answer Robots.txt is a plain text file at yoursite.com/robots.txt that asks crawlers which URLs they may visit. It controls crawling, not indexing: Google says it is not a way to keep a page out of search. It is a request, not a lock, so private pages need a password and pages you want out of Google need noindex.
Want help applying this to your business? Talk it through on a free 30-minute call.
Book a callWhat a robots.txt file is
It's a short text file that sits at the top of your domain, like example.com/robots.txt, and tells search engine crawlers (the programs that fetch web pages, such as Googlebot) which parts of the site they may visit. Anyone can read it, including you: type your domain plus /robots.txt into a browser.
Its real job is managing crawler traffic. Google's Search Central puts the limit plainly:
"This is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google." (Google Search Central: Introduction to robots.txt)
The format is old. Martijn Koster defined it in 1994, and the IETF published it as RFC 9309, a standards-track document, in September 2022 (RFC 9309).
What does a robots.txt file look like?
A few lines, each a field name, a colon and a value. Here is a typical small-business file, line by line:
| Line | What it says |
|---|---|
| User-agent: * | The rules below are for every crawler |
| Disallow: /cart/ | Don't fetch anything whose path starts with /cart/ |
| Disallow: /*?sort= | Don't fetch sorted copies of listing pages |
| Allow: /cart/help | Except this one page, which may be fetched |
| Sitemap: https://example.com/sitemap.xml | Where the list of your pages lives |
Google reads only four fields; anything else, such as crawl-delay, is ignored (Google: robots.txt specification):
| Field | Meaning | Notes |
|---|---|---|
| user-agent | Which crawler the group of rules is for | * means any crawler |
| disallow | A path the crawler should not fetch | A disallow with no path blocks nothing |
| allow | A path it may fetch, even inside a blocked folder | Used to carve exceptions |
| sitemap | Full URL of a sitemap | Applies to all crawlers; you can list several |
Blocking image, script or style files is allowed, but Google warns not to block ones it needs to understand the page (Google Search Central).
A line starting with # is a comment that crawlers skip, handy for noting why a rule exists. With no rules at all, everything may be crawled (Google: robots.txt specification).
How do the rules match a page?
Each rule is compared with the start of a page's path, the part of the address after the domain. Three details trip people up (Google: robots.txt specification):
- Paths are case-sensitive and start with /. Disallow: /fish blocks /fish.html and /fishheads, but not /Fish.asp or /catfish.
- Only two wildcards exist. * stands for any run of characters and $ marks the end of the address. There are no regular expressions (pattern code programmers use elsewhere).
- The longest matching rule wins. If an allow and a disallow match equally, Google picks the less restrictive one, the allow.
| Rule | Blocks | Doesn't block |
|---|---|---|
| Disallow: / | The whole site | Nothing |
| Disallow: /fish/ | /fish/salmon.htm | /fish.html, /animals/fish/ |
| Disallow: /*.pdf$ | /menu.pdf | /menu.pdf?v=2 |
| Disallow: | Nothing (empty rule) | Everything |
Which crawler follows which rules?
Each crawler obeys one group only: the one naming it most specifically. Google says "Other groups are ignored" (Google: robots.txt specification). So if you add a User-agent: Googlebot group, Googlebot stops reading your User-agent: * group entirely. Copy any shared rules into the Googlebot group too. Order of groups in the file doesn't matter.
What robots.txt can't do
This is where most owners go wrong. The file is a polite request that well-behaved crawlers follow. It changes nothing else.
| You want to | Does robots.txt do it? | Use instead |
|---|---|---|
| Keep a page out of Google results | No. A blocked URL can still show up, without a description, if other sites link to it (Google) | A noindex tag (an X-Robots-Tag header for PDFs), a password, or delete the page (Google) |
| Hide private files | No. Listing a path in a public file advertises it (RFC 9309) | A password or login |
| Stop scrapers and bad bots | No. "it's up to the crawler to obey them" (Google) | Server or firewall blocking |
| Slow Googlebot down | No. Google ignores crawl-delay, and Search Console's crawl rate tool was removed January 8, 2024 (Google Blog) | Briefly return 500, 503 or 429 status codes (Google) |
| Cover blog.example.com or the http version | No. Each host, protocol and port needs its own file (Google) | One file per subdomain |
The standard itself is blunt:
"These rules are not a form of access authorization." (RFC 9309)
One trap follows from this. If you block a page in robots.txt and also tag it noindex, Google never fetches the page, so it never sees the noindex: "the crawler will never see the noindex rule, and the page can still appear in search results" (Google Search Central: Block indexing). To take a page off Google, leave it crawlable and let noindex do the work.
Do I need a robots.txt file?
Not necessarily. If the file is missing (the server answers 404), Google treats the site as having no crawl limits and crawls normally (Google: robots.txt specification). Most small sites lose nothing without one. Hosted builders such as Wix or Blogger may manage the file for you and offer a search settings page instead (Google Search Central).
What does hurt is a broken file. If the server returns an error (5xx) for robots.txt, Google stops crawling the site for the first 12 hours, then uses the last good copy for up to 30 days (Google: robots.txt specification).
Where it lives and when changes take effect
- Location: the root of the host only. A file at example.com/folder/robots.txt is never read (Google).
- Name and format: exactly robots.txt, plain UTF-8 text, one file per site (Google).
- Size: Google reads the first 500 KiB and ignores the rest (Google).
- Delay: Google generally caches the file for up to 24 hours, so edits aren't instant (Google).
- Checking it: Google's tool is the robots.txt report in Search Console (Google). See the Search Console reports worth reading.
Example
A landscaper whose file says Disallow: /
The site went live from a staging copy that told every crawler to stay out. Leads from search sit at zero. Deleting that one line, then waiting about a day for Google to refetch the file, lets crawling resume.
A clinic hiding its staff portal
The owner adds Disallow: /staff-portal/. That keeps Google from crawling it but publishes the folder name to anyone who opens robots.txt. The portal needs a login; the robots.txt line adds nothing.
A store with 2,000 filter pages
Sorting and filter options create thousands of near-copies of each category page. A rule like Disallow: /*?sort= keeps crawlers on the real pages. This is the job robots.txt is built for.
FAQ
Does every website have a robots.txt file?
No. It's optional, and with no file, crawlers may fetch any page (RFC 9309).
Can robots.txt remove a page from Google?
No. It stops crawling, not indexing. Use noindex or the steps in removing information from Google.
Can I block AI bots with robots.txt?
Yes, the ones that obey it, by naming each bot's user-agent. Which to block is covered in Should you block AI crawlers?.
How long until Google sees my change?
Usually within a day, since Google caches the file for up to 24 hours (Google).
Different for you?
- Your site isn't in Google at all: a Disallow: / is one of several causes. Work through why a website isn't showing on Google.
- Search Console says pages are crawled but not indexed: that's not a robots.txt problem. See crawled or discovered, currently not indexed.
- You want AI assistants to cite you but not train on you: see blocking AI crawlers.
- You'd rather someone else keep crawl settings right: see /seo/.