Robots.txt setup: create, upload and test your file step by step
Short answer Write robots.txt in any plain text editor, add a User-agent line and your Allow or Disallow rules, then put it at the root of your domain, such as example.com/robots.txt. Open that address in a private window, then check it in Search Console's robots.txt report. Most small sites need three lines, or no custom file at all.
Want help applying this to your business? Talk it through on a free 30-minute call.
Book a callHow to create a robots.txt file: the four steps
Google's documentation splits the job into four steps: create the file, add rules, upload it to the root of your site, and test it (Google Crawling Infrastructure: How to write and submit a robots.txt file).
| Step | What to do | Where it goes wrong |
|---|---|---|
| 1. Create | Open Notepad or TextEdit, save as robots.txt in UTF-8 | A word processor adds curly quotes and hidden formatting |
| 2. Add rules | One User-agent line, then Allow or Disallow lines | A stray "Disallow: /" blocks the whole site |
| 3. Upload | Put it at the top level of the domain | A file in a subfolder is ignored |
| 4. Test | Open it in a private window, then check Search Console | Testing on the wrong version (http vs https, www vs not) |
First, check whether you need to do anything. With no robots.txt file, crawlers may visit everything: Google says that unless the file says otherwise, "all files are implicitly allowed for crawling." Many site builders also write the file for you.
What should a robots.txt file contain?
For most service business sites, three lines are enough. This is the shape Google uses in its own simple example:
| Line | Meaning |
|---|---|
| User-agent: * | The rules below apply to every crawler |
| Allow: / | Every page may be crawled |
| Sitemap: https://www.example.com/sitemap.xml | Where your list of pages lives |
Add a Disallow line only for pages with no value in search, such as an internal search results page or a checkout. The rules that matter, from Google's specification:
- Each group starts with User-agent. That names the bot. An asterisk means all bots, with one catch: it doesn't cover Google's AdsBot crawlers, which "must be named explicitly."
- Paths start with a slash. A folder ends with one too: Disallow: /checkout/.
- Case counts in paths. Google's example: a rule for /file.asp doesn't apply to /FILE.asp.
- The more specific rule wins. When an Allow and a Disallow both match, Google applies the longer path, and on a tie, "the least restrictive rule."
- Two wildcards. An asterisk matches any characters; a dollar sign marks the end of the URL. Google's example Disallow: /*.gif$ stops crawling of every GIF.
- Sitemap needs the full address, including https:// and www if you use it.
- A # starts a comment. Crawlers ignore the rest of that line.
Google reads only four fields: user-agent, allow, disallow and sitemap. It says "other fields such as crawl-delay aren't supported." Some other crawlers do read crawl-delay, but it won't slow Google down.
Where do I put the robots.txt file?
At the root of the exact host your visitors use: https://www.example.com/robots.txt. Google is strict about this:
"The robots.txt file must be located at the root of the site host to which it applies." (Google Crawling Infrastructure: How to write and submit a robots.txt file)
Each host needs its own file. Rules at https://example.com/robots.txt don't cover a subdomain like shop.example.com, and a site can have only one robots.txt per host. If you can't reach the root of your site, Google suggests asking your host or using page-level meta tags instead.
How to edit robots.txt on WordPress, Shopify or Wix
Most owners never upload a file by hand. The platform either writes one or gives you an editor.
| Platform | Default | How to change it |
|---|---|---|
| WordPress (self-hosted) | WordPress serves a virtual file that blocks the admin area | Yoast SEO: Tools, then File Editor. Or upload a real robots.txt to the site root; WordPress shows the virtual one only when no real file exists (Yoast SEO Help) |
| Shopify | A default file that "works for most stores" | Online Store, theme, Edit code, Add a new template, robots. Shopify Support won't help with edits |
| Wix | All pages open to bots by default | SEO & GEO dashboard, Tools and settings, Robots.txt Editor. A Reset to Default button undoes changes |
| Other builders | Varies | Google notes some hosts don't let you edit the file; look for a search visibility setting instead |
Sources: WordPress Developer Resources (do_robots), Yoast SEO Help, Shopify Help Center, Wix Help Center.
One WordPress detail trips people up. The "Discourage search engines" box no longer adds "Disallow: /" to robots.txt. Since WordPress 5.3 it adds a noindex meta tag to pages instead (WordPress Developer Resources: do_robots). So if your site vanished from Google, check that box as well as the file.
Shopify warns that a bad edit "can result in loss of all traffic." Edit the template rather than replacing it with plain text, so Shopify can keep its default rules up to date.
How do I test my robots.txt file?
Test in three passes, from simplest to most exact.
- Open it in a private browser window at yourdomain.com/robots.txt. If you see your rules, the file is public. Use the exact version your visitors see (https, with or without www); each version is its own host with its own file.
- Search Console's robots.txt report. Google names it as the place to fix robots.txt problems, for files already live on your site. It holds the copy of the file Google has, and the Request a recrawl button. See Google Search Console for a small business to set it up.
- Check single pages in Bing's robots.txt Tester. In Bing Webmaster Tools, you enter a URL and the tool acts as Bingbot would, showing whether that URL is allowed or blocked (Bing Blog, September 2020). You can edit rules in the tool and retest before you upload anything.
How long until Google sees my changes?
Up to about a day. Google "generally caches the contents of robots.txt file for up to 24 hours," and longer if it can't fetch a fresh copy (Google: robots.txt specification). For an urgent fix, the robots.txt report has a Request a recrawl option (Google Crawling Infrastructure: Update your robots.txt file).
Your server's answer matters as much as the file:
| Server returns | Google does |
|---|---|
| 200 with the file | Follows the rules |
| 404 or other 4xx (not 429) | Treats it as no file: crawls everything |
| 5xx error | Stops crawling for 12 hours, then uses the last good copy for up to 30 days |
| More than five redirects | Treats it as a 404 |
Google ignores anything past 500 kibibytes (KiB), far more than a small site will ever use.
Common robots.txt mistakes
- "Disallow: /" left over from building the site. It blocks every page for the bots it names. Check for it first when a site won't show on Google; see website not showing up on Google.
- Using Disallow to hide a page. Google says a blocked page "can still be indexed if linked to from other sites." Use noindex or a password instead, and don't block a noindex page, or Google never reads the tag.
- Blocking CSS or script files the page needs. Google warns against this if it makes the page harder for its crawler to understand.
- Treating it as security. The IETF standard says the rules "are not a form of access authorization." Bad bots can ignore them.
- Blocking AI search bots by accident when you meant to block only training bots. See should you block AI crawlers?
Example
A cleaning company on WordPress
The site shows the virtual file WordPress creates: admin area blocked, everything else open. The owner adds one Sitemap line through Yoast's File Editor, opens the file in a private window, and finds no errors in the Search Console report. Total time: 10 minutes.
A new landscaping site that never appeared on Google
The developer left "User-agent: *" and "Disallow: /" from the build. The owner deletes the Disallow line, clicks Request a recrawl, and Google picks up the new file within a day.
A Shopify store with an internal search page
Crawlers fetch thousands of search result URLs. The store adds a Disallow rule for that path in robots.txt.liquid, then checks a product URL and a search URL in Bing's tester: the product is allowed, the search page blocked.
FAQ
Do I need a robots.txt file at all?
No. Without one, crawlers may visit every page, which is what most small businesses want. A file still helps if you want to list your sitemap or keep bots out of pages like checkout.
Can I submit robots.txt to Google?
There's nothing to submit. Google finds the file at your domain root on its own; use Request a recrawl in the robots.txt report only after a change you need picked up fast.
Does robots.txt remove a page from Google?
No. It stops crawling, not indexing, so a blocked URL can still show without a description. To remove a page, use noindex or the steps in removing information from Google.
What's the maximum robots.txt file size?
Google reads the first 500 KiB and ignores the rest. The IETF standard sets that as the minimum any crawler must read.
Does Google follow crawl-delay?
No. Google lists crawl-delay as unsupported. If Googlebot is overloading your server, Google points to its separate guide on reducing crawl rate.
Different for you?
- Search Console says "blocked by robots.txt" or "crawled, currently not indexed": page not indexed in Search Console.
- You want ChatGPT and Perplexity to find you but not train on your pages: should you block AI crawlers?
- Your platform won't let you edit the file and you're thinking of moving: Is Wix or Squarespace bad for SEO?
- You'd rather have crawling and indexing set up and checked for you: SEO & GEO Agent.