Products SEO & GEO Agent Paid Ads Agent Website Agent Conversion Agent
How it works Pricing FAQ Blog Book a call

Reddit's robots.txt block: what it means for your website

Updated · 5 min read

Short answer In July 2024 Reddit changed its robots.txt to tell every crawler to stay out, and let in only companies that signed its terms, Google chief among them. Bing stopped crawling Reddit. Reddit can do that because AI companies pay for its data. A small business needs the opposite: make sure your own robots.txt isn't shutting search engines out by mistake.

Want help applying this to your business? Talk it through on a free 30-minute call.

Book a call

What Reddit blocked, and why

Since July 2024, Reddit's robots.txt tells every crawler to stay out; only companies with a deal get in. Robots.txt is the small text file at yoursite.com/robots.txt that tells search engine bots (crawlers) which pages they may visit.

DateWhat happened
February 22, 2024Reddit gives Google structured access to its posts, including "more efficient ways to train models" (Reddit)
May 16, 2024OpenAI signs a similar deal to bring Reddit content to ChatGPT (Reddit)
June 25, 2024Reddit announces a stricter robots.txt and says bots without an agreement will be rate-limited or blocked (TechCrunch)
July 1, 2024The new file goes live; Bing stops crawling Reddit (Microsoft, via Android Authority)
August 2025Reddit limits the Internet Archive too, saying AI firms were scraping Reddit through it (SiliconANGLE)

Microsoft's statement, as reported: "Bing stopped crawling Reddit after they implemented their updated robots.txt file on July 1," a file "which prohibits all crawling of their site" (Android Authority).

Reddit's reason was AI, not search. A spokesperson said the block was "not at all related to our recent partnership with Google," and that some search engines were "unable or unwilling to make enforceable promises" about using its content, "including their use for AI" (Android Authority). Bing, DuckDuckGo, Mojeek and Qwant stopped showing recent Reddit posts (9to5Google).

Is Reddit really blocking all search engines?

No. Google kept crawling, because companies with an agreement were exempt (TechCrunch). The file says "everyone out"; the deals and the blocking around it decide who actually gets in. We found no 2025–2026 report of Bing getting its access back, but we couldn't confirm today's file either.

Why Reddit can do this and you can't

Reddit sells its content. Its posts are worth enough that Google and OpenAI pay for access (Reddit, Reddit). Closing the door raises the price.

A plumber, dentist or accountant sells a service. Your pages exist to be found. Every crawler you shut out is a search engine or assistant that can't send you a customer. Copy Reddit's file and you lose; copy its habit of deciding on purpose who gets in, and you win.

Lesson 1: check you aren't blocking search engines by accident

Open yoursite.com/robots.txt in a browser. If you see these two lines together, every crawler is told to skip your whole site:

LineMeaning
User-agent: *The next rule is for all bots
Disallow: /Don't crawl any page

Two common ways it happens:

  • A site launched from a test copy. Developers block bots on the test site, then forget to remove the rule at launch.
  • A WordPress checkbox. "Discourage search engines from indexing this site" under Settings, then Reading. Older versions wrote exactly the rule above into robots.txt; since version 5.3 it adds a noindex tag instead (WordPress). Either way, your pages stop showing normally on Google.

After you fix it, Google refreshes its copy within about 24 hours; the Request a recrawl button in Search Console's robots.txt report is faster (Google).

Lesson 2: robots.txt is a request, not a lock

Reddit backs its file with rate limits, blocks and contracts, because the file alone stops nothing. Google says so:

"The instructions in robots.txt files cannot enforce crawler behavior to your site; it's up to the crawler to obey them." (Google Search Central)
"This is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google." (Google Search Central)
You want toUse
Keep a page out of GoogleA noindex tag, with the page left open in robots.txt
Keep files privateA password
Stop a specific AI botA Disallow line for that bot (see which AI bots to block)

Don't block a page in robots.txt and add noindex to it. Google can't read the tag on a page it may not visit:

"If the page is blocked by a robots.txt file or the crawler can't access the page, the crawler will never see the noindex rule, and the page can still appear in search results, for example if other pages link to it." (Google Search Central)

Lesson 3: a broken robots.txt can stop Google too

If your server returns an error (a 5xx code) when Google asks for robots.txt, Google stops crawling your site for the first 12 hours, then uses the last good copy for up to 30 days (Google). A missing file (404) is fine: Google treats it as no restrictions (Google). If your host has outages, ask it to make sure robots.txt loads.

Do I need to manage crawl budget?

Almost certainly not. Google's crawl budget guide is for sites with 1 million+ pages that change weekly, or 10,000+ pages that change daily. Google says if your pages "seem to be crawled the same day that they are published, you don't need to read this guide" (Google). A service business with 30 pages gains nothing by blocking pages to "save" crawling.

Example

A landscaper after a redesign

Calls from Google fall by half in the month after a new site launches. The robots.txt still says User-agent: * and Disallow: /, copied from the test site. Deleting the line and pressing Request a recrawl lets Google back in; pages can come back as Google crawls them again.

A clinic hiding a staff page

The staff login page shows up on Google. The web designer had blocked it in robots.txt, so Google never saw the noindex tag. The fix: unblock it so noindex works, and put the page behind a password.

A bookkeeper who wants out of AI training

She disallows training-only bots and leaves Googlebot, Bingbot and every search bot alone. Like Reddit, she chooses who gets in; unlike Reddit, she keeps every engine that sends clients.

FAQ

Did Reddit disappear from Google?

No. Google was exempt because of its agreement with Reddit (TechCrunch). Bing and engines that rely on it lost recent Reddit posts (9to5Google).

Can a site serve one robots.txt to Google and another to Bing?

Technically yes, but a small business has no reason to. One file with a named rule per bot is clearer, and you can see exactly what every bot is told.

How do I test my robots.txt?

Open it in a private browser window to confirm it loads, then check Search Console's robots.txt report for errors (Google). The old robots.txt Tester was retired in late 2023, as reported by Search Engine Land.

Should I block CSS and JavaScript files?

No. Google says not to block those resource files, or "Google won't do a good job of analyzing pages that depend on those resources" (Google Search Central).

Different for you?

Keep learning

NextRobots.txt vs noindex: which one to use for each pageRobots.txt stops Google crawling a page; noindex keeps it out of results. Which to use for thank-you, ad, search, staging and private pages.

Do it yourself

Crawlability checklist: make every page reachable by GoogleA crawlability checklist for a small business site: find orphan pages, robots.txt blocks, broken links, redirect chains and server errors, with free tools.

Ask us one question

One line is enough. We reply by email.

Sources

  1. TechCrunch: Reddit's upcoming changes attempt to safeguard the platform against AI crawlers (June 25, 2024)
  2. Android Authority: Microsoft confirms Reddit blocked Bing (July 25, 2024), quoting Microsoft and Reddit
  3. 9to5Google: Reddit search engine block and the Google deal (July 25, 2024)
  4. Reddit: Expanding our Partnership with Google (February 22, 2024)
  5. Reddit: Reddit and OpenAI Build Partnership (May 16, 2024)
  6. SiliconANGLE: Reddit limits the Internet Archive (August 11, 2025)
  7. Google Search Central: Introduction to robots.txt
  8. Google Crawling: How Google interprets the robots.txt specification
  9. Google Crawling: Update your robots.txt file
  10. Google Search Central: Block Search indexing with noindex
  11. Google Crawling: Crawl budget management for large sites
  12. Search Engine Land: Google Search Console adds robots.txt report (November 2023)
  13. WordPress Documentation: Settings Reading Screen