Reddit's robots.txt block: what it means for your website
Short answer In July 2024 Reddit changed its robots.txt to tell every crawler to stay out, and let in only companies that signed its terms, Google chief among them. Bing stopped crawling Reddit. Reddit can do that because AI companies pay for its data. A small business needs the opposite: make sure your own robots.txt isn't shutting search engines out by mistake.
Want help applying this to your business? Talk it through on a free 30-minute call.
Book a callWhat Reddit blocked, and why
Since July 2024, Reddit's robots.txt tells every crawler to stay out; only companies with a deal get in. Robots.txt is the small text file at yoursite.com/robots.txt that tells search engine bots (crawlers) which pages they may visit.
| Date | What happened |
|---|---|
| February 22, 2024 | Reddit gives Google structured access to its posts, including "more efficient ways to train models" (Reddit) |
| May 16, 2024 | OpenAI signs a similar deal to bring Reddit content to ChatGPT (Reddit) |
| June 25, 2024 | Reddit announces a stricter robots.txt and says bots without an agreement will be rate-limited or blocked (TechCrunch) |
| July 1, 2024 | The new file goes live; Bing stops crawling Reddit (Microsoft, via Android Authority) |
| August 2025 | Reddit limits the Internet Archive too, saying AI firms were scraping Reddit through it (SiliconANGLE) |
Microsoft's statement, as reported: "Bing stopped crawling Reddit after they implemented their updated robots.txt file on July 1," a file "which prohibits all crawling of their site" (Android Authority).
Reddit's reason was AI, not search. A spokesperson said the block was "not at all related to our recent partnership with Google," and that some search engines were "unable or unwilling to make enforceable promises" about using its content, "including their use for AI" (Android Authority). Bing, DuckDuckGo, Mojeek and Qwant stopped showing recent Reddit posts (9to5Google).
Is Reddit really blocking all search engines?
No. Google kept crawling, because companies with an agreement were exempt (TechCrunch). The file says "everyone out"; the deals and the blocking around it decide who actually gets in. We found no 2025–2026 report of Bing getting its access back, but we couldn't confirm today's file either.
Why Reddit can do this and you can't
Reddit sells its content. Its posts are worth enough that Google and OpenAI pay for access (Reddit, Reddit). Closing the door raises the price.
A plumber, dentist or accountant sells a service. Your pages exist to be found. Every crawler you shut out is a search engine or assistant that can't send you a customer. Copy Reddit's file and you lose; copy its habit of deciding on purpose who gets in, and you win.
Lesson 1: check you aren't blocking search engines by accident
Open yoursite.com/robots.txt in a browser. If you see these two lines together, every crawler is told to skip your whole site:
| Line | Meaning |
|---|---|
| User-agent: * | The next rule is for all bots |
| Disallow: / | Don't crawl any page |
Two common ways it happens:
- A site launched from a test copy. Developers block bots on the test site, then forget to remove the rule at launch.
- A WordPress checkbox. "Discourage search engines from indexing this site" under Settings, then Reading. Older versions wrote exactly the rule above into robots.txt; since version 5.3 it adds a noindex tag instead (WordPress). Either way, your pages stop showing normally on Google.
After you fix it, Google refreshes its copy within about 24 hours; the Request a recrawl button in Search Console's robots.txt report is faster (Google).
Lesson 2: robots.txt is a request, not a lock
Reddit backs its file with rate limits, blocks and contracts, because the file alone stops nothing. Google says so:
"The instructions in robots.txt files cannot enforce crawler behavior to your site; it's up to the crawler to obey them." (Google Search Central)
"This is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google." (Google Search Central)
| You want to | Use |
|---|---|
| Keep a page out of Google | A noindex tag, with the page left open in robots.txt |
| Keep files private | A password |
| Stop a specific AI bot | A Disallow line for that bot (see which AI bots to block) |
Don't block a page in robots.txt and add noindex to it. Google can't read the tag on a page it may not visit:
"If the page is blocked by a robots.txt file or the crawler can't access the page, the crawler will never see the noindex rule, and the page can still appear in search results, for example if other pages link to it." (Google Search Central)
Lesson 3: a broken robots.txt can stop Google too
If your server returns an error (a 5xx code) when Google asks for robots.txt, Google stops crawling your site for the first 12 hours, then uses the last good copy for up to 30 days (Google). A missing file (404) is fine: Google treats it as no restrictions (Google). If your host has outages, ask it to make sure robots.txt loads.
Do I need to manage crawl budget?
Almost certainly not. Google's crawl budget guide is for sites with 1 million+ pages that change weekly, or 10,000+ pages that change daily. Google says if your pages "seem to be crawled the same day that they are published, you don't need to read this guide" (Google). A service business with 30 pages gains nothing by blocking pages to "save" crawling.
Example
A landscaper after a redesign
Calls from Google fall by half in the month after a new site launches. The robots.txt still says User-agent: * and Disallow: /, copied from the test site. Deleting the line and pressing Request a recrawl lets Google back in; pages can come back as Google crawls them again.
A clinic hiding a staff page
The staff login page shows up on Google. The web designer had blocked it in robots.txt, so Google never saw the noindex tag. The fix: unblock it so noindex works, and put the page behind a password.
A bookkeeper who wants out of AI training
She disallows training-only bots and leaves Googlebot, Bingbot and every search bot alone. Like Reddit, she chooses who gets in; unlike Reddit, she keeps every engine that sends clients.
FAQ
Did Reddit disappear from Google?
No. Google was exempt because of its agreement with Reddit (TechCrunch). Bing and engines that rely on it lost recent Reddit posts (9to5Google).
Can a site serve one robots.txt to Google and another to Bing?
Technically yes, but a small business has no reason to. One file with a named rule per bot is clearer, and you can see exactly what every bot is told.
How do I test my robots.txt?
Open it in a private browser window to confirm it loads, then check Search Console's robots.txt report for errors (Google). The old robots.txt Tester was retired in late 2023, as reported by Search Engine Land.
Should I block CSS and JavaScript files?
No. Google says not to block those resource files, or "Google won't do a good job of analyzing pages that depend on those resources" (Google Search Central).
Different for you?
- You want to keep AI training bots out but stay in AI answers: should you block AI crawlers?
- Your site isn't on Google at all: website not showing up on Google.
- Search Console says pages are crawled but not indexed: crawled or discovered, currently not indexed.
- You've never opened Search Console: Google Search Console for a small business.
- You want customers from Reddit itself: Reddit marketing for a small business.
- You'd rather have it checked for you: SEO & GEO Agent.