Screaming Frog for content syndication: check partner pages
Short answer Paste the partner pages that republished your article into Screaming Frog's List mode, free for up to 500 addresses. Read the Directives tab: each copy should show noindex, which Google recommends for syndicated copies. Then check the copy links to your original. To pick posts to share, crawl your blog and keep indexable pages that already earn search clicks.
Want help applying this to your business? Talk it through on a free 30-minute call.
Book a callWhat should a partner's copy of your article show?
Two things: a noindex tag, so Google leaves their copy out of search, and a visible link to your original. Content syndication means another site republishes an article you already posted. The risk isn't a penalty: Google wrote that unless duplication is meant to deceive, "it's unlikely that one of those ways will be a penalty" (Google Search Central Blog, 2008). The risk is that a bigger site's copy shows in search instead of yours.
Google's current advice settles which tag the partner should use:
"The canonical link element is not recommended for those who want to avoid duplication by syndication partners, because the pages are often very different. The most effective solution is for partners to block indexing of your content." (Google Search Central: Fix canonicalization issues)
A canonical tag is a line in a page's code naming the "main" version of that page. Many syndication checklists still tell you to ask for one pointing at your site. Google no longer recommends it for this job. Noindex is the tag that tells search engines not to list a page. Google says it can sit in the page's code as a robots meta tag or in the server's response as an X-Robots-Tag header (Google Search Central: Block Search indexing with noindex).
| On the partner's copy | Good | Problem |
|---|---|---|
| Noindex | Present | Missing: their copy can compete with yours |
| Canonical | Pointing at your original, or absent, as long as noindex is there | Pointing at their own page with no noindex |
| Link to you | A working link to your original article | No link, or a link to your homepage only |
| Page status | Loads (code 200) | Redirects or errors: the article was moved or taken down |
Do you need Screaming Frog for this?
Not for one or two partners: open their page in a browser, view the source and search for "noindex" and "canonical". Screaming Frog's SEO Spider, a desktop program that reads pages the way Google's crawler does, earns its place when you have a dozen copies or more, or recheck them every few months.
| Free | Paid licence | |
|---|---|---|
| Price | $0 | £199 a year per user; $279 in US dollars, November 2026 |
| Addresses per crawl | 500 | No set limit |
| List mode, Directives and Canonicals tabs | Yes | Yes |
| Outlinks for each page | Yes | Yes |
| Custom search ("does this page contain my link?") | No | Yes |
| JavaScript rendering | No | Yes |
| Search Console and Analytics data | No | Yes |
| Save crawls, compare two crawls, schedule | No | Yes |
Prices and paid-only features are from Screaming Frog's pricing page, November 2026. It shows prices in pounds, US dollars and euros; there is no Canadian-dollar price.
How to check partner pages, step by step
These steps follow Screaming Frog's user guide as of November 2026.
- List every copy. Keep a sheet of each partner and the exact address of their copy. Ask the partner if you don't have it.
- Switch to List mode. Choose Mode > List. The user guide says you can then "either paste or upload a list of specific URLs to crawl". Paste your list and click Start. The free version takes up to 500 addresses.
- Read Response Codes. Anything other than 200 means the copy moved or is gone. Many 403 or 429 codes mean the site is refusing the crawler; check those pages by hand.
- Read the Directives tab, Noindex filter. The tab "shows data related to the meta robots tag, and the X-Robots-Tag in the HTTP Header", so it catches both ways of setting noindex. Every copy should be listed here.
- Read the Canonicals tab. "Canonicalised" means the page names another address as the main one; check it is yours. "Self Referencing" means the page names itself.
- Check the link back. Click a copy in the top window, then the Outlinks tab at the bottom, and look for your article's address. With a licence, Configuration > Custom > Search does this in bulk: set it to find your address and filter pages that "does not contain" it.
- Export and file. Export each tab as a spreadsheet and note the date. The free version can't save a crawl, so the export is your record.
Two limits. The free version reads the page's code as delivered, without running JavaScript; if a partner's site adds tags with JavaScript, confirm by hand. And Screaming Frog obeys robots.txt, the file that tells crawlers which pages to skip, so a partner can block it.
What the results mean, and what to ask the partner
Most problems come down to one email asking for noindex.
| What you see | What it means | What to ask |
|---|---|---|
| Noindex present, link to your article | Done right | Nothing |
| No noindex, canonical to your article | Older setup Google doesn't recommend for syndication | Add noindex |
| No noindex, canonical to their own page | Their copy can rank in place of yours | Add noindex |
| Page blocked by robots.txt instead of noindex | Google can still list it | Unblock and add noindex |
| No link to you | Readers can't find your site | Add an "originally published at" link |
The robots.txt row surprises people. Google says that if a page is blocked by robots.txt, "the crawler will never see the noindex rule, and the page can still appear in search results, for example if other pages link to it" (Google Search Central: Block Search indexing with noindex).
If a partner won't add noindex, check whether their copy already shows instead of yours. In Search Console, the URL Inspection tool on your own article shows which page Google treats as the main one (Google Search Central: Fix canonicalization issues). How to read that report: Page indexing and URL Inspection.
Which of your posts should you syndicate?
Share posts Google already has indexed and that already earn clicks. If Google hasn't settled on your original yet, a copy on a larger site has a better chance of being shown first. Use Screaming Frog on your own blog to filter, then your search data to rank.
- Crawl your blog. Enter your blog's address and click Start.
- Keep indexable pages. In the Internal tab, sort by Indexability. Drop anything marked Non-Indexable; the Indexability Status column gives the reason, such as "canonicalised to another URL".
- Keep self-referencing canonicals. In the Canonicals tab, your originals should sit under Self Referencing, naming themselves as the main version.
- Fix broken links first. Response Codes, Client Error (4xx), shows links that go nowhere. A partner copies your article with its links; fix them before you send it.
- Rank by search clicks. Open Search Console's Performance report, Pages tab, and sort by clicks. The paid licence can pull these numbers into the crawl; the free route is Search Console's own export (Search Console for a small business).
- Favor lasting topics. A checklist or how-to stays useful on a partner site for years; a news post doesn't.
Example
Illustrative cases with invented numbers.
A landscaping company in Oregon (US dollars)
The owner's spring lawn guide runs on four regional home and garden sites. He pastes the four addresses into the free version. Two show noindex and a link back. One has a canonical to his page but no noindex; one has neither. He emails both editors asking for noindex. Cost: $0.
A software reseller with 40 copies (US dollars)
The marketing lead has 40 republished articles across 12 sites. She buys a $279 licence, sets Custom Search to find her domain, and lists 6 copies with no link back. Next quarter she reruns the saved crawl and compares the two.
An accounting firm in Toronto (Canadian dollars)
The firm crawls its blog with the free version: 180 pages. It keeps 120 indexable posts with self-referencing canonicals, then picks the 5 with the most clicks in Search Console. It sends those 5 to a provincial business association's newsletter site, asking for noindex and a link. Cost: $0.
FAQ
Should the partner use canonical or noindex?
Noindex. Google says the canonical tag is not recommended for syndication and that the best fix is for partners to block indexing of your content.
Will syndicated copies get my site penalized?
Not for ordinary republishing with credit. The practical risk is the partner's copy showing in search in place of your original.
Can I crawl a partner's site without asking?
Screaming Frog obeys robots.txt by default, and the free version uses 5 parallel requests. Crawling only the few pages that carry your article, in List mode, keeps the load small.
Does Google News treat syndicated copies differently?
Yes. A noindex aimed only at Googlebot-News keeps a copy out of Google News but not Search. Ask for plain noindex (Google News Publisher Center Help).
Different for you?
- Someone copied your article without permission: duplicate content.
- You'd rather turn one post into emails, videos and social posts yourself: repurpose content.
- You want to see who links to your original after it's republished: backlink monitoring.
- You're new to the tool and want a first crawl explained: Screaming Frog competitor analysis.
- You'd rather have syndication checked and partners chased for you: SEO & GEO Agent.