Products SEO & GEO Agent Paid Ads Agent Website Agent Conversion Agent
How it works Pricing FAQ Blog Book a call

Screaming Frog skyscraper technique: find content to beat

Updated · 7 min read

Short answer Screaming Frog does two of the skyscraper jobs: it lists a rival's pages with titles, headings and internal links, and it checks a list of linking pages in bulk. It can't tell you who links to a rival. That list comes from a link database such as Ahrefs, Majestic or Moz, which plugs into the paid licence.

Want help applying this to your business? Talk it through on a free 30-minute call.

Book a call

What Screaming Frog does in a skyscraper campaign, and what it doesn't

The skyscraper technique, as Backlinko named it in 2013, has three steps: "Find link-worthy content", "Make something even better", "Reach out to the right people". Screaming Frog's SEO Spider is a crawler, a desktop program that follows links on a website and lists every page it reaches. That makes it good at the first step's homework and at checking lists. It is not a backlink tool.

Skyscraper jobScreaming Frog alone (free)Needs something else
List a rival's articles with titles, headings, word countYes, up to 500 URLs per crawl
Show which pages the rival pushes with its own linksYes (internal links)
Show how many other sites link to each pageNoLicence plus Ahrefs, Majestic or Moz, or a backlink checker
List the sites that link to the winning pageNoA backlink checker
Check that those linking pages still work and still linkYes, in list mode
Write the better page and send emailsNoYou

The free version stops at 500 URLs per crawl; a licence costs "£199 a year" (Screaming Frog user guide: General), shown as US$279 on its pricing page in November 2026. No Canadian-dollar price is listed.

Step 1: crawl the rival's content

Start where the articles live, not at the homepage, so the 500-URL limit goes to the pages you care about.

  1. Install the SEO Spider (Windows, macOS or Ubuntu Linux) and type the rival's blog or resources address, such as theirsite.com/blog/, into the box at the top. Click Start.
  2. Open the Internal tab and pick the HTML filter. Each row is one page.
  3. Look at four columns: Title 1, H1-1, Word Count and Unique Inlinks. Unique Inlinks counts how many of the rival's own pages link to that page.
  4. Sort by Unique Inlinks. Pages near the top are the ones the rival treats as important. Click Export to save the list as a spreadsheet.

Internal links tell you what the rival values, not what the web values. A guide with 80 internal links may have zero links from other sites. Step 2 checks that.

You need a link count from outside the site. There are two ways.

With a licence. Under Configuration > API Access you can connect Ahrefs, Majestic or Moz. The data then appears in the Link Metrics tab against each URL you crawl (Screaming Frog user guide: Tabs). Each still needs its own account: Ahrefs needs "a paid subscription", Majestic needs "a subscription which allows you to pull data from their API", and Moz offers "a free limited API" (Screaming Frog user guide: Configuration). Crawl the rival's blog, sort by referring domains (the number of different websites linking to a page), and the link magnets rise to the top.

Without a licence. Copy the top 10 to 20 URLs from your Step 1 export and paste each into a backlink checker. Ahrefs' free Backlink Checker calls itself "a glimpse into the power of our premium tool". Its Starter plan ($29 a month) allows 250 rows per report and no exports (Ahrefs Help: Starter plan).

Pick the page with the most referring domains on a topic you can genuinely cover better. A page linked from 40 different sites is a better target than one linked 400 times from three.

This list is the heart of the technique, and only a link database has it. Enter the winning page's address in the backlink tool's backlinks or referring domains report and copy the linking pages into a spreadsheet: page address, site name, and the link's anchor text (the clickable words).

Screaming Frog's link-metrics integration fills in counts per URL; for the names of the linking sites, use the backlink tool's own report.

Step 4: clean the list with a list-mode crawl

Link databases keep links that have since been removed. Screaming Frog checks them in one pass.

  1. Switch Mode to List, paste the linking page addresses (up to 500 free) and click Start.
  2. In Response Codes, drop pages that return 4xx or 5xx errors. Those pages are gone.
  3. Run Bulk Export > Links > All Outlinks. Filter the link destinations for the rival's page. Linking pages missing from the result no longer link to it.

With a licence, Custom Search (Configuration > Custom > Search) can instead flag every crawled page whose HTML contains the rival's address.

What is left is your outreach list: live pages that still link to the page you plan to beat.

The External tab is not your outreach list

A common mistake is to crawl the rival's article, open the External tab and treat those sites as prospects. That tab lists URLs "on a different subdomain" than the crawl's start page (Screaming Frog user guide: Tabs): sites the rival links to, not sites that link to the rival. Those are the rival's sources. They may be worth reading, but they have shown no interest in the rival's page.

Build something better: what "better" means

Better means more useful to the people who linked, not longer. Google says plainly:

"The length of the content alone doesn't matter for ranking purposes (there's no magical word count target, minimum or maximum, though you probably want to have at least one word)." (Google Search Central: SEO Starter Guide)

Use the Word Count column to judge depth, not as a target. Then look at why the linkers linked. Read five linking pages: do they cite a number, a checklist, a cost table? Give them a fresher or more complete version of that thing. Your own prices, photos and job data are what a rival can't copy. See content that earns backlinks for what makes people link.

Pitch without breaking Google's rules

Asking a site to swap a link to your better page is fine. Paying for it is not. Google's spam policy lists, as link spam:

"Buying or selling links for ranking purposes." (Google Search Central: Spam policies for Google web search)

The same list includes trading goods or services for links and excessive "link to me and I'll link to you" swaps. Write each email to the person who published the page, say what changed since the page they link to was written, and offer nothing in return. For wording that doesn't read as spam, see AI outreach for link pitches.

Is it worth it for a small business?

It depends on whether your topic has pages with many links. National topics (a cost guide, a checklist, a data study) often do. A local plumber's topics often don't, and the relationship links in backlinks for a local business come easier.

SetupCostFits
Free crawl plus free backlink checker$0One small campaign, rival blog under 500 URLs
Free crawl plus Ahrefs Starter for one month$29250 rows per report is enough for one target page
Licence plus a link-data accountUS$279 a year plus the link toolSeveral campaigns a year, large rival sites

Example

Illustrative cases with invented numbers.

A home inspector in Ohio (US dollars)

A national inspection site's blog has 300 URLs, so the free crawl covers it. Its "home inspection checklist" has the most internal links. The free backlink checker shows it is linked by realtor blogs. She writes a checklist from her own reports, finds 25 linking pages, and the list-mode crawl drops 6 dead ones. She emails 19 and gets 2 links. Cost: $0.

A bookkeeper in Toronto

A rival's "small business tax deadlines" page leads on internal links, but a month of Ahrefs Starter (US$29) shows only 4 sites link to it. A "bookkeeping cost" page with fewer internal links has 30. He builds the better cost page instead.

A landscaper who stopped early

The rival's top pages have no outside links at all. He skips the technique and asks his suppliers and local associations for links instead.

FAQ

Can Screaming Frog show who links to a competitor?

No. It sees only links on pages it crawls. Who links to a page comes from a link database such as Ahrefs, Majestic or Moz.

Do I need the paid licence for the skyscraper technique?

Not for one small campaign. The free 500-URL crawl covers most small rival blogs and outreach lists. The licence adds link-data integrations and custom search.

Should I switch to database storage first?

It is already the default: the SEO Spider "will start in database storage mode". The setting is under Settings > Storage Mode.

How do I know it worked?

Re-check your new page in the same backlink checker after a month or two: count new referring domains. Watch its clicks in Google Search Console.

Is the skyscraper technique against Google's rules?

No, if you only ask. Offering money, goods or services for the link is link spam under Google's policies.

Different for you?

Keep learning

NextScreaming Frog for guest posting: vet a prospect listHow to use Screaming Frog's free crawler to check guest post sites before you pitch: link sellers, nofollow rules, relevance and contact pages.

Ask us one question

One line is enough. We reply by email.

Sources

  1. Backlinko: The Skyscraper Technique
  2. Screaming Frog: SEO Spider user guide, General
  3. Screaming Frog: SEO Spider user guide, Configuration
  4. Screaming Frog: SEO Spider user guide, Tabs
  5. Screaming Frog: SEO Spider pricing
  6. Ahrefs: Free Backlink Checker
  7. Ahrefs Help: About Ahrefs Starter plan
  8. Google Search Central: SEO Starter Guide
  9. Google Search Central: Spam policies for Google web search