Products SEO & GEO Agent Paid Ads Agent Website Agent Conversion Agent
How it works Pricing FAQ Blog Book a call

Robots.txt setup: create, upload and test your file step by step

Updated · 7 min read

Short answer Write robots.txt in any plain text editor, add a User-agent line and your Allow or Disallow rules, then put it at the root of your domain, such as example.com/robots.txt. Open that address in a private window, then check it in Search Console's robots.txt report. Most small sites need three lines, or no custom file at all.

Want help applying this to your business? Talk it through on a free 30-minute call.

Book a call

How to create a robots.txt file: the four steps

Google's documentation splits the job into four steps: create the file, add rules, upload it to the root of your site, and test it (Google Crawling Infrastructure: How to write and submit a robots.txt file).

StepWhat to doWhere it goes wrong
1. CreateOpen Notepad or TextEdit, save as robots.txt in UTF-8A word processor adds curly quotes and hidden formatting
2. Add rulesOne User-agent line, then Allow or Disallow linesA stray "Disallow: /" blocks the whole site
3. UploadPut it at the top level of the domainA file in a subfolder is ignored
4. TestOpen it in a private window, then check Search ConsoleTesting on the wrong version (http vs https, www vs not)

First, check whether you need to do anything. With no robots.txt file, crawlers may visit everything: Google says that unless the file says otherwise, "all files are implicitly allowed for crawling." Many site builders also write the file for you.

What should a robots.txt file contain?

For most service business sites, three lines are enough. This is the shape Google uses in its own simple example:

LineMeaning
User-agent: *The rules below apply to every crawler
Allow: /Every page may be crawled
Sitemap: https://www.example.com/sitemap.xmlWhere your list of pages lives

Add a Disallow line only for pages with no value in search, such as an internal search results page or a checkout. The rules that matter, from Google's specification:

  • Each group starts with User-agent. That names the bot. An asterisk means all bots, with one catch: it doesn't cover Google's AdsBot crawlers, which "must be named explicitly."
  • Paths start with a slash. A folder ends with one too: Disallow: /checkout/.
  • Case counts in paths. Google's example: a rule for /file.asp doesn't apply to /FILE.asp.
  • The more specific rule wins. When an Allow and a Disallow both match, Google applies the longer path, and on a tie, "the least restrictive rule."
  • Two wildcards. An asterisk matches any characters; a dollar sign marks the end of the URL. Google's example Disallow: /*.gif$ stops crawling of every GIF.
  • Sitemap needs the full address, including https:// and www if you use it.
  • A # starts a comment. Crawlers ignore the rest of that line.

Google reads only four fields: user-agent, allow, disallow and sitemap. It says "other fields such as crawl-delay aren't supported." Some other crawlers do read crawl-delay, but it won't slow Google down.

Where do I put the robots.txt file?

At the root of the exact host your visitors use: https://www.example.com/robots.txt. Google is strict about this:

"The robots.txt file must be located at the root of the site host to which it applies." (Google Crawling Infrastructure: How to write and submit a robots.txt file)

Each host needs its own file. Rules at https://example.com/robots.txt don't cover a subdomain like shop.example.com, and a site can have only one robots.txt per host. If you can't reach the root of your site, Google suggests asking your host or using page-level meta tags instead.

How to edit robots.txt on WordPress, Shopify or Wix

Most owners never upload a file by hand. The platform either writes one or gives you an editor.

PlatformDefaultHow to change it
WordPress (self-hosted)WordPress serves a virtual file that blocks the admin areaYoast SEO: Tools, then File Editor. Or upload a real robots.txt to the site root; WordPress shows the virtual one only when no real file exists (Yoast SEO Help)
ShopifyA default file that "works for most stores"Online Store, theme, Edit code, Add a new template, robots. Shopify Support won't help with edits
WixAll pages open to bots by defaultSEO & GEO dashboard, Tools and settings, Robots.txt Editor. A Reset to Default button undoes changes
Other buildersVariesGoogle notes some hosts don't let you edit the file; look for a search visibility setting instead

Sources: WordPress Developer Resources (do_robots), Yoast SEO Help, Shopify Help Center, Wix Help Center.

One WordPress detail trips people up. The "Discourage search engines" box no longer adds "Disallow: /" to robots.txt. Since WordPress 5.3 it adds a noindex meta tag to pages instead (WordPress Developer Resources: do_robots). So if your site vanished from Google, check that box as well as the file.

Shopify warns that a bad edit "can result in loss of all traffic." Edit the template rather than replacing it with plain text, so Shopify can keep its default rules up to date.

How do I test my robots.txt file?

Test in three passes, from simplest to most exact.

  1. Open it in a private browser window at yourdomain.com/robots.txt. If you see your rules, the file is public. Use the exact version your visitors see (https, with or without www); each version is its own host with its own file.
  2. Search Console's robots.txt report. Google names it as the place to fix robots.txt problems, for files already live on your site. It holds the copy of the file Google has, and the Request a recrawl button. See Google Search Console for a small business to set it up.
  3. Check single pages in Bing's robots.txt Tester. In Bing Webmaster Tools, you enter a URL and the tool acts as Bingbot would, showing whether that URL is allowed or blocked (Bing Blog, September 2020). You can edit rules in the tool and retest before you upload anything.

How long until Google sees my changes?

Up to about a day. Google "generally caches the contents of robots.txt file for up to 24 hours," and longer if it can't fetch a fresh copy (Google: robots.txt specification). For an urgent fix, the robots.txt report has a Request a recrawl option (Google Crawling Infrastructure: Update your robots.txt file).

Your server's answer matters as much as the file:

Server returnsGoogle does
200 with the fileFollows the rules
404 or other 4xx (not 429)Treats it as no file: crawls everything
5xx errorStops crawling for 12 hours, then uses the last good copy for up to 30 days
More than five redirectsTreats it as a 404

Google ignores anything past 500 kibibytes (KiB), far more than a small site will ever use.

Common robots.txt mistakes

  • "Disallow: /" left over from building the site. It blocks every page for the bots it names. Check for it first when a site won't show on Google; see website not showing up on Google.
  • Using Disallow to hide a page. Google says a blocked page "can still be indexed if linked to from other sites." Use noindex or a password instead, and don't block a noindex page, or Google never reads the tag.
  • Blocking CSS or script files the page needs. Google warns against this if it makes the page harder for its crawler to understand.
  • Treating it as security. The IETF standard says the rules "are not a form of access authorization." Bad bots can ignore them.
  • Blocking AI search bots by accident when you meant to block only training bots. See should you block AI crawlers?

Example

A cleaning company on WordPress

The site shows the virtual file WordPress creates: admin area blocked, everything else open. The owner adds one Sitemap line through Yoast's File Editor, opens the file in a private window, and finds no errors in the Search Console report. Total time: 10 minutes.

A new landscaping site that never appeared on Google

The developer left "User-agent: *" and "Disallow: /" from the build. The owner deletes the Disallow line, clicks Request a recrawl, and Google picks up the new file within a day.

A Shopify store with an internal search page

Crawlers fetch thousands of search result URLs. The store adds a Disallow rule for that path in robots.txt.liquid, then checks a product URL and a search URL in Bing's tester: the product is allowed, the search page blocked.

FAQ

Do I need a robots.txt file at all?

No. Without one, crawlers may visit every page, which is what most small businesses want. A file still helps if you want to list your sitemap or keep bots out of pages like checkout.

Can I submit robots.txt to Google?

There's nothing to submit. Google finds the file at your domain root on its own; use Request a recrawl in the robots.txt report only after a change you need picked up fast.

Does robots.txt remove a page from Google?

No. It stops crawling, not indexing, so a blocked URL can still show without a description. To remove a page, use noindex or the steps in removing information from Google.

What's the maximum robots.txt file size?

Google reads the first 500 KiB and ignores the rest. The IETF standard sets that as the minimum any crawler must read.

Does Google follow crawl-delay?

No. Google lists crawl-delay as unsupported. If Googlebot is overloading your server, Google points to its separate guide on reducing crawl rate.

Different for you?

Keep learning

NextRobots.txt Mistakes That Block Google, and How to Fix ThemThe robots.txt errors that quietly hide a site from Google: a leftover staging block, blocked CSS, loose wildcards, dead rules. How to spot and fix each.

Do it yourself

Crawlability checklist: make every page reachable by GoogleA crawlability checklist for a small business site: find orphan pages, robots.txt blocks, broken links, redirect chains and server errors, with free tools.

Ask us one question

One line is enough. We reply by email.

Sources

  1. Google Crawling Infrastructure: How to write and submit a robots.txt file
  2. Google Crawling Infrastructure: How Google interprets the robots.txt specification
  3. Google Crawling Infrastructure: Update your robots.txt file
  4. Google Crawling Infrastructure: Useful robots.txt rules
  5. Google Search Central: Block Search indexing with noindex
  6. Google Search Central: Introduction to robots.txt
  7. RFC 9309: Robots Exclusion Protocol (IETF, September 2022)
  8. Bing Blog: Bing Webmaster Tools makes it easy to edit and verify your robots.txt (September 2020)
  9. WordPress Developer Resources: do_robots()
  10. Yoast SEO Help: How to edit robots.txt through Yoast SEO
  11. Shopify Help Center: Editing robots.txt.liquid
  12. Wix Help Center: Editing your site's robots.txt file