Screaming Frog skyscraper technique: find content to beat
Short answer Screaming Frog does two of the skyscraper jobs: it lists a rival's pages with titles, headings and internal links, and it checks a list of linking pages in bulk. It can't tell you who links to a rival. That list comes from a link database such as Ahrefs, Majestic or Moz, which plugs into the paid licence.
Want help applying this to your business? Talk it through on a free 30-minute call.
Book a callWhat Screaming Frog does in a skyscraper campaign, and what it doesn't
The skyscraper technique, as Backlinko named it in 2013, has three steps: "Find link-worthy content", "Make something even better", "Reach out to the right people". Screaming Frog's SEO Spider is a crawler, a desktop program that follows links on a website and lists every page it reaches. That makes it good at the first step's homework and at checking lists. It is not a backlink tool.
| Skyscraper job | Screaming Frog alone (free) | Needs something else |
|---|---|---|
| List a rival's articles with titles, headings, word count | Yes, up to 500 URLs per crawl | |
| Show which pages the rival pushes with its own links | Yes (internal links) | |
| Show how many other sites link to each page | No | Licence plus Ahrefs, Majestic or Moz, or a backlink checker |
| List the sites that link to the winning page | No | A backlink checker |
| Check that those linking pages still work and still link | Yes, in list mode | |
| Write the better page and send emails | No | You |
The free version stops at 500 URLs per crawl; a licence costs "£199 a year" (Screaming Frog user guide: General), shown as US$279 on its pricing page in November 2026. No Canadian-dollar price is listed.
Step 1: crawl the rival's content
Start where the articles live, not at the homepage, so the 500-URL limit goes to the pages you care about.
- Install the SEO Spider (Windows, macOS or Ubuntu Linux) and type the rival's blog or resources address, such as theirsite.com/blog/, into the box at the top. Click Start.
- Open the Internal tab and pick the HTML filter. Each row is one page.
- Look at four columns: Title 1, H1-1, Word Count and Unique Inlinks. Unique Inlinks counts how many of the rival's own pages link to that page.
- Sort by Unique Inlinks. Pages near the top are the ones the rival treats as important. Click Export to save the list as a spreadsheet.
Internal links tell you what the rival values, not what the web values. A guide with 80 internal links may have zero links from other sites. Step 2 checks that.
Step 2: find which pages actually earned links
You need a link count from outside the site. There are two ways.
With a licence. Under Configuration > API Access you can connect Ahrefs, Majestic or Moz. The data then appears in the Link Metrics tab against each URL you crawl (Screaming Frog user guide: Tabs). Each still needs its own account: Ahrefs needs "a paid subscription", Majestic needs "a subscription which allows you to pull data from their API", and Moz offers "a free limited API" (Screaming Frog user guide: Configuration). Crawl the rival's blog, sort by referring domains (the number of different websites linking to a page), and the link magnets rise to the top.
Without a licence. Copy the top 10 to 20 URLs from your Step 1 export and paste each into a backlink checker. Ahrefs' free Backlink Checker calls itself "a glimpse into the power of our premium tool". Its Starter plan ($29 a month) allows 250 rows per report and no exports (Ahrefs Help: Starter plan).
Pick the page with the most referring domains on a topic you can genuinely cover better. A page linked from 40 different sites is a better target than one linked 400 times from three.
Step 3: list who links to the winning page
This list is the heart of the technique, and only a link database has it. Enter the winning page's address in the backlink tool's backlinks or referring domains report and copy the linking pages into a spreadsheet: page address, site name, and the link's anchor text (the clickable words).
Screaming Frog's link-metrics integration fills in counts per URL; for the names of the linking sites, use the backlink tool's own report.
Step 4: clean the list with a list-mode crawl
Link databases keep links that have since been removed. Screaming Frog checks them in one pass.
- Switch Mode to List, paste the linking page addresses (up to 500 free) and click Start.
- In Response Codes, drop pages that return 4xx or 5xx errors. Those pages are gone.
- Run Bulk Export > Links > All Outlinks. Filter the link destinations for the rival's page. Linking pages missing from the result no longer link to it.
With a licence, Custom Search (Configuration > Custom > Search) can instead flag every crawled page whose HTML contains the rival's address.
What is left is your outreach list: live pages that still link to the page you plan to beat.
The External tab is not your outreach list
A common mistake is to crawl the rival's article, open the External tab and treat those sites as prospects. That tab lists URLs "on a different subdomain" than the crawl's start page (Screaming Frog user guide: Tabs): sites the rival links to, not sites that link to the rival. Those are the rival's sources. They may be worth reading, but they have shown no interest in the rival's page.
Build something better: what "better" means
Better means more useful to the people who linked, not longer. Google says plainly:
"The length of the content alone doesn't matter for ranking purposes (there's no magical word count target, minimum or maximum, though you probably want to have at least one word)." (Google Search Central: SEO Starter Guide)
Use the Word Count column to judge depth, not as a target. Then look at why the linkers linked. Read five linking pages: do they cite a number, a checklist, a cost table? Give them a fresher or more complete version of that thing. Your own prices, photos and job data are what a rival can't copy. See content that earns backlinks for what makes people link.
Pitch without breaking Google's rules
Asking a site to swap a link to your better page is fine. Paying for it is not. Google's spam policy lists, as link spam:
"Buying or selling links for ranking purposes." (Google Search Central: Spam policies for Google web search)
The same list includes trading goods or services for links and excessive "link to me and I'll link to you" swaps. Write each email to the person who published the page, say what changed since the page they link to was written, and offer nothing in return. For wording that doesn't read as spam, see AI outreach for link pitches.
Is it worth it for a small business?
It depends on whether your topic has pages with many links. National topics (a cost guide, a checklist, a data study) often do. A local plumber's topics often don't, and the relationship links in backlinks for a local business come easier.
| Setup | Cost | Fits |
|---|---|---|
| Free crawl plus free backlink checker | $0 | One small campaign, rival blog under 500 URLs |
| Free crawl plus Ahrefs Starter for one month | $29 | 250 rows per report is enough for one target page |
| Licence plus a link-data account | US$279 a year plus the link tool | Several campaigns a year, large rival sites |
Example
Illustrative cases with invented numbers.
A home inspector in Ohio (US dollars)
A national inspection site's blog has 300 URLs, so the free crawl covers it. Its "home inspection checklist" has the most internal links. The free backlink checker shows it is linked by realtor blogs. She writes a checklist from her own reports, finds 25 linking pages, and the list-mode crawl drops 6 dead ones. She emails 19 and gets 2 links. Cost: $0.
A bookkeeper in Toronto
A rival's "small business tax deadlines" page leads on internal links, but a month of Ahrefs Starter (US$29) shows only 4 sites link to it. A "bookkeeping cost" page with fewer internal links has 30. He builds the better cost page instead.
A landscaper who stopped early
The rival's top pages have no outside links at all. He skips the technique and asks his suppliers and local associations for links instead.
FAQ
Can Screaming Frog show who links to a competitor?
No. It sees only links on pages it crawls. Who links to a page comes from a link database such as Ahrefs, Majestic or Moz.
Do I need the paid licence for the skyscraper technique?
Not for one small campaign. The free 500-URL crawl covers most small rival blogs and outreach lists. The licence adds link-data integrations and custom search.
Should I switch to database storage first?
It is already the default: the SEO Spider "will start in database storage mode". The setting is under Settings > Storage Mode.
How do I know it worked?
Re-check your new page in the same backlink checker after a month or two: count new referring domains. Watch its clicks in Google Search Console.
Is the skyscraper technique against Google's rules?
No, if you only ask. Offering money, goods or services for the link is link spam under Google's policies.
Different for you?
- You want the full Screaming Frog walkthrough for a rival site: Screaming Frog competitor analysis.
- The rival's best page is dead, not live: broken link building.
- You're a local business with few linkable topics: backlinks for a local business.
- You need to decide what makes a page worth linking to: content that earns backlinks.
- You'd rather have the research and outreach done for you: SEO & GEO Agent.