Screaming Frog for guest posting: vet a prospect list
Short answer Screaming Frog doesn't find guest post sites; it checks sites you already found. Paste each prospect's homepage and two or three of its guest posts into List mode (free up to 500 URLs). Read the Outlinks tab: keyword-stuffed links to unrelated businesses mean the site sells links. Drop those, keep relevant sites, and pitch through their contact page.
Want help applying this to your business? Talk it through on a free 30-minute call.
Book a callWhat Screaming Frog can and can't do for guest posting
It's a checker, not a finder. Screaming Frog's SEO Spider is a crawler: a desktop program that visits the addresses you give it and records what's on each page, including every link out. It has no list of websites that accept guest articles. You bring the list; it tells you which sites on it are worth an email.
| Job | Screaming Frog? | Free version? |
|---|---|---|
| Find sites in your field that publish outside writers | No. Use Google searches and seed pages (link prospecting) | n/a |
| Crawl a pasted list of prospect pages | Yes, List mode | Yes, up to 500 URLs |
| See every link out of a guest post, its anchor text (the clickable words) and whether it's nofollow | Yes, Outlinks tab | Yes |
| Spot broken pages and missing titles on a prospect | Yes | Yes |
| Search every page for "write for us" or "sponsored post" | Yes, Custom Search | No, licence only |
| Pull author names or contact details into columns | Yes, Custom Extraction | No, licence only |
| Add Ahrefs, Majestic or Moz link scores | Yes, API Access | No; also needs that tool's account |
The free version is enough for a list of 50 to 100 prospects. Screaming Frog's pricing page lists configuration, saving crawls and custom search and extraction as licence features; the licence is £199 a year per user (pricing page, November 2026).
Is guest posting for links still allowed by Google?
Writing for another site is fine. Doing it to plant keyword links is spam. Google's spam policies list this among link schemes:
"Advertorials or native advertising where payment is received for articles that include links that pass ranking credit, or links with optimized anchor text in articles, guest posts, or press releases distributed on other sites." (Google Search Central: Spam policies for Google web search)
Google's 2017 reminder on article campaigns names the warning signs, including "Having the articles published across many different sites" and "Stuffing keyword-rich links to your site in your articles." It also says sites that publish such articles can be hurt:
"When Google detects that a website is publishing articles that contain spammy links, this may change Google's perception of the quality of the site and could affect its ranking." (Google Search Central Blog, 2017)
That's the real reason to vet. A site that sells links is a site Google may already distrust, and your link there is worth little or worse. Pitch a guest post for the readers and the referral visits, not for a ranking boost; see effective link building.
Step 1: build the starting list outside the tool
Collect 20 to 50 candidate sites from:
- Google searches such as `"write for us" landscaping` or `"guest post" bookkeeping`.
- Trade associations, suppliers and local publications that already feature businesses like yours (link prospecting).
- Sites that link to your competitors, from a link tool (backlink analysis).
For each site, copy three addresses into a text file: the homepage and two recent articles by outside authors (look for a "guest author" byline or a contributor page).
Step 2: crawl the list in List mode
- Download the SEO Spider and open it. It runs on Windows, Mac and Ubuntu Linux.
- Choose Mode > List, click Upload, and paste your addresses. Include https:// on each.
- Click Start. In List mode the crawl depth is set to 0, so "only the URLs uploaded will be crawled" (Screaming Frog: How to use List mode). Three pages for 50 sites is 150 URLs, well under the free cap.
- Keep the default speed. The tool crawls with 5 threads "to not overload servers" and obeys each site's robots.txt (the file that tells crawlers what to skip). If a site returns 403 (forbidden) or 429 (too many requests), it's refusing crawlers. Check it by hand; don't disguise the crawler.
Step 3: vet each prospect from the crawl
Click a guest article in the top window, then open the Outlinks tab at the bottom. It lists every link on that page, with Anchor Text, a Follow column ("False" means the link carries nofollow, UGC or sponsored) and a Rel column.
| What you see | What it means | Keep? |
|---|---|---|
| Two or three links out, in plain words, to relevant sources | Edited, normal publishing | Yes |
| Author links marked nofollow or sponsored | The site follows Google's rules; your link won't pass ranking credit but can send visitors | Yes, if its readers are your customers |
| Many links per article with exact keyword anchors ("best cheap roofing Dallas") to unrelated businesses | Link selling | No |
| Article topics jump between casinos, CBD, crypto and plumbing | No real audience; built to sell links | No |
| Page titles and headings (Page Titles and H1 tabs) match your field | Relevant readers | Yes |
| Many 404 errors on pages you crawled | Poorly kept site | Lower on the list |
Google's own rule for publishers explains the nofollow result: links that are "advertisements or paid placements" should carry rel="sponsored" (Google Search Central: Qualify your outbound links). A site that tags guest links honestly is a better partner than one that doesn't. See nofollow vs dofollow.
To keep the list, click the Export button next to Upload and Start: it saves the crawl to a spreadsheet in the order you uploaded it (Screaming Frog: How to use List mode). Add two columns, Keep and Contact page, and fill them as you go.
Third-party scores help sort, not decide. Moz says its Domain Authority "is not a direct ranking factor" (Moz: What is Domain Authority?). A small trade blog your customers read beats a high-score site that publishes anything.
Step 4: find the editor's contact page
Crawl one shortlisted site in normal Spider mode (paste the homepage, click Start). Type `contact`, `write` or `contribute` into the search box at the top right; it filters the Address column. Open the page and read the submission rules before you write.
Use the address or form the site gives for pitches. Scraping every email address out of a site is a legal risk:
- US. The FTC's CAN-SPAM guide lists "harvesting email addresses" among acts that carry criminal penalties, and says "The law makes no exception for business-to-business email." A commercial email needs your postal address and a way to opt out.
- Canada. CASL needs consent for a commercial email. A published business address counts only if the publication "is not accompanied by a statement that the person does not wish to receive unsolicited commercial electronic messages" and the message is relevant to the person's role (CASL, section 10(9)(b)). A "write for us" page invites pitches; a staff list with "no solicitations" does not.
More on both: cold outreach within spam law and CASL consent.
Do you need the paid licence?
Not for a small business pitching a few sites a month. The free version covers List mode, the Outlinks tab and 500 URLs per crawl. A licence saves time only when you vet hundreds of sites and want Custom Search to flag "sponsored post" pages automatically or saved crawls to re-check placed links later.
Example
Illustrative cases with invented numbers.
A Toronto bookkeeper (Canadian dollars)
She finds 30 small-business blogs, crawls 90 URLs free, and drops 11 whose guest posts carry three or more keyword links to unrelated sites. Of the 19 left, six list a pitch email on a contributor page. Cost: $0 and one afternoon.
A Phoenix pool cleaner (US dollars)
A site offers a "guest post" for $150. The crawl shows its articles link to a casino, a loan broker and a roofer, each with exact keyword anchors. He skips it: under Google's policy that's a paid link.
A dental clinic
A regional health magazine marks all author links sponsored. The clinic pitches anyway: the readers are local patients, and visits count even without ranking credit.
FAQ
Can Screaming Frog find guest posting sites for me?
No. It crawls addresses you supply. Find candidates with Google searches, industry associations and a backlink tool, then crawl them to check quality.
Is Screaming Frog free for this?
Yes, for up to 500 URLs per crawl. Custom Search, Custom Extraction and API link scores need the £199-a-year licence.
Should I change the user-agent to get past a site that blocks the crawler?
No. A block means the site doesn't want crawlers. Check those pages in your browser; changing the user-agent also requires a licence.
Do nofollow guest post links help?
Google treats nofollow, sponsored and ugc as "hints about which links to consider or exclude within Search" (Google Search Central Blog, 2019), so expect little ranking credit. They still bring readers who click.
Different for you?
- You have no prospects yet: link prospecting.
- You want to know which links are worth getting: quality backlinks.
- You're writing the pitch email: AI outreach.
- You want to crawl a rival's site instead: Screaming Frog competitor analysis.
- You'd rather have prospecting and outreach done for you: SEO & GEO Agent.