Products SEO & GEO Agent Paid Ads Agent Website Agent Conversion Agent
How it works Pricing FAQ Blog Book a call

Screaming Frog for content syndication: check partner pages

Updated · 7 min read

Short answer Paste the partner pages that republished your article into Screaming Frog's List mode, free for up to 500 addresses. Read the Directives tab: each copy should show noindex, which Google recommends for syndicated copies. Then check the copy links to your original. To pick posts to share, crawl your blog and keep indexable pages that already earn search clicks.

Want help applying this to your business? Talk it through on a free 30-minute call.

Book a call

What should a partner's copy of your article show?

Two things: a noindex tag, so Google leaves their copy out of search, and a visible link to your original. Content syndication means another site republishes an article you already posted. The risk isn't a penalty: Google wrote that unless duplication is meant to deceive, "it's unlikely that one of those ways will be a penalty" (Google Search Central Blog, 2008). The risk is that a bigger site's copy shows in search instead of yours.

Google's current advice settles which tag the partner should use:

"The canonical link element is not recommended for those who want to avoid duplication by syndication partners, because the pages are often very different. The most effective solution is for partners to block indexing of your content." (Google Search Central: Fix canonicalization issues)

A canonical tag is a line in a page's code naming the "main" version of that page. Many syndication checklists still tell you to ask for one pointing at your site. Google no longer recommends it for this job. Noindex is the tag that tells search engines not to list a page. Google says it can sit in the page's code as a robots meta tag or in the server's response as an X-Robots-Tag header (Google Search Central: Block Search indexing with noindex).

On the partner's copyGoodProblem
NoindexPresentMissing: their copy can compete with yours
CanonicalPointing at your original, or absent, as long as noindex is therePointing at their own page with no noindex
Link to youA working link to your original articleNo link, or a link to your homepage only
Page statusLoads (code 200)Redirects or errors: the article was moved or taken down

Do you need Screaming Frog for this?

Not for one or two partners: open their page in a browser, view the source and search for "noindex" and "canonical". Screaming Frog's SEO Spider, a desktop program that reads pages the way Google's crawler does, earns its place when you have a dozen copies or more, or recheck them every few months.

FreePaid licence
Price$0£199 a year per user; $279 in US dollars, November 2026
Addresses per crawl500No set limit
List mode, Directives and Canonicals tabsYesYes
Outlinks for each pageYesYes
Custom search ("does this page contain my link?")NoYes
JavaScript renderingNoYes
Search Console and Analytics dataNoYes
Save crawls, compare two crawls, scheduleNoYes

Prices and paid-only features are from Screaming Frog's pricing page, November 2026. It shows prices in pounds, US dollars and euros; there is no Canadian-dollar price.

How to check partner pages, step by step

These steps follow Screaming Frog's user guide as of November 2026.

  1. List every copy. Keep a sheet of each partner and the exact address of their copy. Ask the partner if you don't have it.
  2. Switch to List mode. Choose Mode > List. The user guide says you can then "either paste or upload a list of specific URLs to crawl". Paste your list and click Start. The free version takes up to 500 addresses.
  3. Read Response Codes. Anything other than 200 means the copy moved or is gone. Many 403 or 429 codes mean the site is refusing the crawler; check those pages by hand.
  4. Read the Directives tab, Noindex filter. The tab "shows data related to the meta robots tag, and the X-Robots-Tag in the HTTP Header", so it catches both ways of setting noindex. Every copy should be listed here.
  5. Read the Canonicals tab. "Canonicalised" means the page names another address as the main one; check it is yours. "Self Referencing" means the page names itself.
  6. Check the link back. Click a copy in the top window, then the Outlinks tab at the bottom, and look for your article's address. With a licence, Configuration > Custom > Search does this in bulk: set it to find your address and filter pages that "does not contain" it.
  7. Export and file. Export each tab as a spreadsheet and note the date. The free version can't save a crawl, so the export is your record.

Two limits. The free version reads the page's code as delivered, without running JavaScript; if a partner's site adds tags with JavaScript, confirm by hand. And Screaming Frog obeys robots.txt, the file that tells crawlers which pages to skip, so a partner can block it.

What the results mean, and what to ask the partner

Most problems come down to one email asking for noindex.

What you seeWhat it meansWhat to ask
Noindex present, link to your articleDone rightNothing
No noindex, canonical to your articleOlder setup Google doesn't recommend for syndicationAdd noindex
No noindex, canonical to their own pageTheir copy can rank in place of yoursAdd noindex
Page blocked by robots.txt instead of noindexGoogle can still list itUnblock and add noindex
No link to youReaders can't find your siteAdd an "originally published at" link

The robots.txt row surprises people. Google says that if a page is blocked by robots.txt, "the crawler will never see the noindex rule, and the page can still appear in search results, for example if other pages link to it" (Google Search Central: Block Search indexing with noindex).

If a partner won't add noindex, check whether their copy already shows instead of yours. In Search Console, the URL Inspection tool on your own article shows which page Google treats as the main one (Google Search Central: Fix canonicalization issues). How to read that report: Page indexing and URL Inspection.

Which of your posts should you syndicate?

Share posts Google already has indexed and that already earn clicks. If Google hasn't settled on your original yet, a copy on a larger site has a better chance of being shown first. Use Screaming Frog on your own blog to filter, then your search data to rank.

  1. Crawl your blog. Enter your blog's address and click Start.
  2. Keep indexable pages. In the Internal tab, sort by Indexability. Drop anything marked Non-Indexable; the Indexability Status column gives the reason, such as "canonicalised to another URL".
  3. Keep self-referencing canonicals. In the Canonicals tab, your originals should sit under Self Referencing, naming themselves as the main version.
  4. Fix broken links first. Response Codes, Client Error (4xx), shows links that go nowhere. A partner copies your article with its links; fix them before you send it.
  5. Rank by search clicks. Open Search Console's Performance report, Pages tab, and sort by clicks. The paid licence can pull these numbers into the crawl; the free route is Search Console's own export (Search Console for a small business).
  6. Favor lasting topics. A checklist or how-to stays useful on a partner site for years; a news post doesn't.

Example

Illustrative cases with invented numbers.

A landscaping company in Oregon (US dollars)

The owner's spring lawn guide runs on four regional home and garden sites. He pastes the four addresses into the free version. Two show noindex and a link back. One has a canonical to his page but no noindex; one has neither. He emails both editors asking for noindex. Cost: $0.

A software reseller with 40 copies (US dollars)

The marketing lead has 40 republished articles across 12 sites. She buys a $279 licence, sets Custom Search to find her domain, and lists 6 copies with no link back. Next quarter she reruns the saved crawl and compares the two.

An accounting firm in Toronto (Canadian dollars)

The firm crawls its blog with the free version: 180 pages. It keeps 120 indexable posts with self-referencing canonicals, then picks the 5 with the most clicks in Search Console. It sends those 5 to a provincial business association's newsletter site, asking for noindex and a link. Cost: $0.

FAQ

Should the partner use canonical or noindex?

Noindex. Google says the canonical tag is not recommended for syndication and that the best fix is for partners to block indexing of your content.

Will syndicated copies get my site penalized?

Not for ordinary republishing with credit. The practical risk is the partner's copy showing in search in place of your original.

Can I crawl a partner's site without asking?

Screaming Frog obeys robots.txt by default, and the free version uses 5 parallel requests. Crawling only the few pages that carry your article, in List mode, keeps the load small.

Does Google News treat syndicated copies differently?

Yes. A noindex aimed only at Googlebot-News keeps a copy out of Google News but not Search. Ask for plain noindex (Google News Publisher Center Help).

Different for you?

Keep learning

NextScreaming Frog broken link building: find dead links to replaceHow to crawl resource pages in Screaming Frog for dead outside links you can replace: list mode settings, reading 4xx results, free vs paid licence.

Ask us one question

One line is enough. We reply by email.

Sources

  1. Google Search Central: Fix canonicalization issues
  2. Google Search Central: Block Search indexing with noindex
  3. Google Search Central Blog: Demystifying the duplicate content penalty (2008)
  4. Google News Publisher Center Help: Avoid article duplication in Google News
  5. Screaming Frog: SEO Spider pricing
  6. Screaming Frog: SEO Spider user guide, General
  7. Screaming Frog: SEO Spider user guide, Tabs
  8. Screaming Frog: SEO Spider user guide, Configuration