Products SEO & GEO Agent Paid Ads Agent Website Agent Conversion Agent
How it works Pricing FAQ Blog Book a call

Screaming Frog competitor analysis: a step-by-step for owners

Updated · 8 min read

Want help applying this to your business? Talk it through on a free 30-minute call.

Book a call

What a crawl of a competitor's site shows, and what it doesn't

It shows how their site is built, not how it performs. Screaming Frog's Search engine optimization: work that helps your pages show up higher in the unpaid results on Google and Bing. More Spider is a crawler: a desktop program that follows the links on a site, as Google's own crawler does, and lists every address it reaches.

You can seeYou can't see
Every page reachable through links, and the folders they sit in (/services/, /locations/)Pages that nothing on the site links to
Each page's title, meta description (the summary under a search result) and headingsHow many visitors each page gets
Word count, and how many of their own links point at each pageWhich searches they rank for, or in what position
Broken links, redirects, and images with no alt text (a written description of the picture)Which other websites link to them (without a link-data add-on)
Their Google Business Profile, reviews or ads

Traffic is the gap people miss. The tool can pull in Google's free tool that shows which searches bring people to your site and flags pages Google can't index. More and Analytics numbers, but only by connecting "a Google account (which has access to the Search Console account you wish to query)", in the words of its user guide. A competitor's numbers stay private.

In local search, the website is only part of the race:

"Local results are mainly based on relevance, distance, and popularity." (Google Business Profile Help: Tips to improve your local ranking on Google)

A rival who beats you on Google Maps may just have more reviews or sit closer to the searcher. No crawl shows that; see how to rank higher on Google Maps.

Free or paid: do you need the licence?

For a one-time look at a few small competitors, no. The free version handles 500 URLs per crawl. On the paid side:

"For £199 a year you can buy a licence, which removes the 500 URL crawl limit." (Screaming Frog SEO Spider user guide: General)
FreePaid licence
Price$0£199 a year per user; the US-dollar price shown in November 2026 was $279
Pages per crawl500Limited only by your computer's memory and disk
Titles, descriptions, headings, broken links, redirects, exact duplicatesYesYes
Change settings (skip images, limit depth, slow the crawl)NoYes
Save a crawl, reopen it, compare two crawlsNoYes
Scheduled crawlsNoYes
Custom extraction (pull any element, such as prices)NoYes
Search Console, Analytics, PageSpeed and backlink dataNoYes; Search Console and Analytics only for accounts you can open

Screaming Frog's pricing page lists US dollars, pounds and euros, with no Canadian-dollar price. A licence lasts one year and covers one person.

The common tip to untick images, CSS and JavaScript before crawling doesn't apply to free users: the user guide says a licence "also provides access to the configuration".

Buy it only if a rival has thousands of pages you need in full, or you'll re-crawl on a schedule. For a rough list of a rival's pages, a Google search for "site:theirsite.com" costs nothing, though Google says the list "is not always exhaustive" (Google Search Central: site: search operator).

How to crawl a competitor's website, step by step

These steps follow Screaming Frog's user guide as of November 2026, free version.

  1. Pick two or three rivals. Search Google for your main service and your town. The websites in the regular results, below the map, are the ones a crawl can teach you about. Skip directories such as Yelp; pick businesses like yours.
  2. Install it. Download the SEO Spider from Screaming Frog's website. It runs on Windows, macOS and Linux (Ubuntu and Fedora) and needs "a 64-bit OS with at least 4gb of RAM available".
  3. Crawl your own site first. Type your homepage into the address box at the top and click Start. This is your baseline.
  4. Export. The Export button at the top left saves the tab you're viewing as a spreadsheet. Do it before the next crawl: saving and reopening crawls needs a licence.
  5. Crawl the rival. Enter their homepage and click Start. By default the tool obeys the site's robots.txt (the file that tells crawlers what to skip) and runs 5 threads (parallel requests) "to not overload servers".
  6. Watch the response codes. Many 403 (forbidden) or 429 (too many requests) codes mean the site is refusing or slowing the crawler. Stop there. With a licence you can slow it to one page per second under Configuration > Speed.
  7. Need one section only? Start at a folder such as /services/ and pick Subfolder in the switcher at the top. It works only if that folder's main page links to the pages inside it. Or switch to List mode and paste the exact addresses.

Which tabs to read, and what they tell you

Each view turns into a question about your own site.

Tab (filter)What to look atWhat it suggests for you
Internal (HTML)Every page and the folders they sit inServices, towns or FAQ pages they have and you lack
Page TitlesTheir pattern: service, town, brand nameHow you word yours. Google calls a good title "unique to the page, clear and concise"
Meta DescriptionOffers, prices, reasons to clickWhat they promise before anyone visits
H1 and H2How a service page is split: cost, process, warranty, questionsQuestions your page should answer too
Internal: Inlinks and Crawl DepthPages most linked from their own site; pages many clicks from the homepageWhich pages they push hardest, and whether yours are buried
Response Codes (Client Error 4xx) and Images (Missing Alt Text)Broken pages and undescribed picturesRun the same check on your own crawl

Three things look important and mostly aren't, according to Google's SEO Starter Guide:

Turning the crawl into a to-do list

Fix your own problems first, then fill real gaps:

  1. Your own errors. Broken links and missing titles in your crawl are the cheapest fixes you'll find. The Issues tab on the right lists them by priority.
  2. Services you offer but never described. If the rival has a drain cleaning page and you only list it, write that page from your own prices, photos and process.
  3. Links to the pages that earn money. If their key service pages are one click from the homepage and yours are three, put yours in the menu.
  4. Not their town-page farm. A crawl often reveals dozens of near-identical pages, one per town. See city pages for service areas.
  5. A date to look again. Re-crawl when a rival starts beating you, or every few months. With a licence, scheduled crawls and crawl comparison do it for you.

Two of Google's rules decide how far to copy. Its quality questions ask:

"If the content draws on other sources, does it avoid simply copying or rewriting those sources, and instead provide substantial additional value and originality?" (Google Search Central: Creating helpful, reliable, people-first content)

And its spam policy lists, as doorway abuse, "Having multiple domain names or pages targeted at specific regions or cities that funnel users to one page" (Google Search Central: Spam policies for Google web search).

For what a crawl can't answer, use free tools:

QuestionFree tool
Which searches show my site?Google Search Console, your own site only (guide)
Which words do customers type?Search Console, autocomplete and Keyword Planner (guide)
Are their pages faster for real visitors?Google's PageSpeed Insights: real users' experience over the past 28 days, when a page has enough visits (guide)
Who links to them?A link database such as Ahrefs, Majestic or Moz, mostly paid; see backlinks for a local business

Example

Illustrative cases with invented numbers.

A roofer and the top result (US dollars)

The rival's site has 180 URLs, so the free crawl covers it. The Internal tab shows pages for metal roofs, storm damage, insurance claims and financing; the roofer has one Services page. Plan: four pages with his own prices and job photos, each in his menu. Tool cost: $0.

A physiotherapy clinic in Calgary (Canadian dollars)

The rival has 2,400 URLs, mostly blog posts, so a free crawl from the homepage stops at 500. Instead of a licence, the owner starts at the rival's /services/ folder and gets all 40 service pages. Three are treatments she offers but never wrote up.

A landscaper facing 70 town pages

Every rival town page has the same text with the town swapped. He doesn't copy it; he builds pages for the five towns where he has real jobs, photos and reviews.

FAQ

Is Screaming Frog free?

Yes, up to 500 URLs per crawl. A licence (£199 or US$279 a year per user, November 2026) adds settings, saved crawls, scheduling and integrations.

Will a competitor know I crawled their site?

Possibly. By default it names itself "Screaming Frog SEO Spider" in every request and obeys their robots.txt; the name can show in their server records.

Can Screaming Frog show a competitor's keywords, traffic or backlinks?

Not alone. It reads Search Console and Analytics only for accounts you can open. Links to your website from other websites. Links from trusted, relevant sites can help your pages rank. More need a licence plus an Ahrefs, Majestic or Moz account.

Different for you?

Keep learning

Ask us one question

One line is enough. We reply by email.

Sources

  1. Screaming Frog: SEO Spider pricing
  2. Screaming Frog: SEO Spider user guide, General
  3. Screaming Frog: SEO Spider user guide, Configuration
  4. Screaming Frog: SEO Spider user guide, Tabs
  5. Google Search Central: SEO Starter Guide
  6. Google Search Central: Creating helpful, reliable, people-first content
  7. Google Search Central: Spam policies for Google web search
  8. Google Search Central: site: search operator
  9. Google Business Profile Help: Tips to improve your local ranking on Google
  10. PageSpeed Insights: About PageSpeed Insights