What a crawl of a competitor's site shows, and what it doesn't
It shows how their site is built, not how it performs. Screaming Frog's Search engine optimization: work that helps your pages show up higher in the unpaid results on Google and Bing. More Spider is a crawler: a desktop program that follows the links on a site, as Google's own crawler does, and lists every address it reaches.
| You can see | You can't see |
|---|---|
| Every page reachable through links, and the folders they sit in (/services/, /locations/) | Pages that nothing on the site links to |
| Each page's title, meta description (the summary under a search result) and headings | How many visitors each page gets |
| Word count, and how many of their own links point at each page | Which searches they rank for, or in what position |
| Broken links, redirects, and images with no alt text (a written description of the picture) | Which other websites link to them (without a link-data add-on) |
| Their Google Business Profile, reviews or ads |
Traffic is the gap people miss. The tool can pull in Google's free tool that shows which searches bring people to your site and flags pages Google can't index. More and Analytics numbers, but only by connecting "a Google account (which has access to the Search Console account you wish to query)", in the words of its user guide. A competitor's numbers stay private.
In local search, the website is only part of the race:
"Local results are mainly based on relevance, distance, and popularity." (Google Business Profile Help: Tips to improve your local ranking on Google)
A rival who beats you on Google Maps may just have more reviews or sit closer to the searcher. No crawl shows that; see how to rank higher on Google Maps.
Free or paid: do you need the licence?
For a one-time look at a few small competitors, no. The free version handles 500 URLs per crawl. On the paid side:
"For £199 a year you can buy a licence, which removes the 500 URL crawl limit." (Screaming Frog SEO Spider user guide: General)
| Free | Paid licence | |
|---|---|---|
| Price | $0 | £199 a year per user; the US-dollar price shown in November 2026 was $279 |
| Pages per crawl | 500 | Limited only by your computer's memory and disk |
| Titles, descriptions, headings, broken links, redirects, exact duplicates | Yes | Yes |
| Change settings (skip images, limit depth, slow the crawl) | No | Yes |
| Save a crawl, reopen it, compare two crawls | No | Yes |
| Scheduled crawls | No | Yes |
| Custom extraction (pull any element, such as prices) | No | Yes |
| Search Console, Analytics, PageSpeed and backlink data | No | Yes; Search Console and Analytics only for accounts you can open |
Screaming Frog's pricing page lists US dollars, pounds and euros, with no Canadian-dollar price. A licence lasts one year and covers one person.
The common tip to untick images, CSS and JavaScript before crawling doesn't apply to free users: the user guide says a licence "also provides access to the configuration".
Buy it only if a rival has thousands of pages you need in full, or you'll re-crawl on a schedule. For a rough list of a rival's pages, a Google search for "site:theirsite.com" costs nothing, though Google says the list "is not always exhaustive" (Google Search Central: site: search operator).
How to crawl a competitor's website, step by step
These steps follow Screaming Frog's user guide as of November 2026, free version.
- Pick two or three rivals. Search Google for your main service and your town. The websites in the regular results, below the map, are the ones a crawl can teach you about. Skip directories such as Yelp; pick businesses like yours.
- Install it. Download the SEO Spider from Screaming Frog's website. It runs on Windows, macOS and Linux (Ubuntu and Fedora) and needs "a 64-bit OS with at least 4gb of RAM available".
- Crawl your own site first. Type your homepage into the address box at the top and click Start. This is your baseline.
- Export. The Export button at the top left saves the tab you're viewing as a spreadsheet. Do it before the next crawl: saving and reopening crawls needs a licence.
- Crawl the rival. Enter their homepage and click Start. By default the tool obeys the site's robots.txt (the file that tells crawlers what to skip) and runs 5 threads (parallel requests) "to not overload servers".
- Watch the response codes. Many 403 (forbidden) or 429 (too many requests) codes mean the site is refusing or slowing the crawler. Stop there. With a licence you can slow it to one page per second under Configuration > Speed.
- Need one section only? Start at a folder such as /services/ and pick Subfolder in the switcher at the top. It works only if that folder's main page links to the pages inside it. Or switch to List mode and paste the exact addresses.
Which tabs to read, and what they tell you
Each view turns into a question about your own site.
| Tab (filter) | What to look at | What it suggests for you |
|---|---|---|
| Internal (HTML) | Every page and the folders they sit in | Services, towns or FAQ pages they have and you lack |
| Page Titles | Their pattern: service, town, brand name | How you word yours. Google calls a good title "unique to the page, clear and concise" |
| Meta Description | Offers, prices, reasons to click | What they promise before anyone visits |
| H1 and H2 | How a service page is split: cost, process, warranty, questions | Questions your page should answer too |
| Internal: Inlinks and Crawl Depth | Pages most linked from their own site; pages many clicks from the homepage | Which pages they push hardest, and whether yours are buried |
| Response Codes (Client Error 4xx) and Images (Missing Alt Text) | Broken pages and undescribed pictures | Run the same check on your own crawl |
Three things look important and mostly aren't, according to Google's SEO Starter Guide:
- Word count. "The length of the content alone doesn't matter for ranking purposes." A longer rival page isn't a length to beat.
- Heading count and order. Order helps screen readers, but "from Google Search perspective, it doesn't matter if you're using them out of order." Screaming Frog flags pages with several H1s, yet its own guide calls that "not strictly an issue".
- Keywords in web addresses. Keywords in the domain or URL path "alone have hardly any effect beyond appearing in breadcrumbs".
Turning the crawl into a to-do list
Fix your own problems first, then fill real gaps:
- Your own errors. Broken links and missing titles in your crawl are the cheapest fixes you'll find. The Issues tab on the right lists them by priority.
- Services you offer but never described. If the rival has a drain cleaning page and you only list it, write that page from your own prices, photos and process.
- Links to the pages that earn money. If their key service pages are one click from the homepage and yours are three, put yours in the menu.
- Not their town-page farm. A crawl often reveals dozens of near-identical pages, one per town. See city pages for service areas.
- A date to look again. Re-crawl when a rival starts beating you, or every few months. With a licence, scheduled crawls and crawl comparison do it for you.
Two of Google's rules decide how far to copy. Its quality questions ask:
"If the content draws on other sources, does it avoid simply copying or rewriting those sources, and instead provide substantial additional value and originality?" (Google Search Central: Creating helpful, reliable, people-first content)
And its spam policy lists, as doorway abuse, "Having multiple domain names or pages targeted at specific regions or cities that funnel users to one page" (Google Search Central: Spam policies for Google web search).
For what a crawl can't answer, use free tools:
| Question | Free tool |
|---|---|
| Which searches show my site? | Google Search Console, your own site only (guide) |
| Which words do customers type? | Search Console, autocomplete and Keyword Planner (guide) |
| Are their pages faster for real visitors? | Google's PageSpeed Insights: real users' experience over the past 28 days, when a page has enough visits (guide) |
| Who links to them? | A link database such as Ahrefs, Majestic or Moz, mostly paid; see backlinks for a local business |
Example
Illustrative cases with invented numbers.
A roofer and the top result (US dollars)
The rival's site has 180 URLs, so the free crawl covers it. The Internal tab shows pages for metal roofs, storm damage, insurance claims and financing; the roofer has one Services page. Plan: four pages with his own prices and job photos, each in his menu. Tool cost: $0.
A physiotherapy clinic in Calgary (Canadian dollars)
The rival has 2,400 URLs, mostly blog posts, so a free crawl from the homepage stops at 500. Instead of a licence, the owner starts at the rival's /services/ folder and gets all 40 service pages. Three are treatments she offers but never wrote up.
A landscaper facing 70 town pages
Every rival town page has the same text with the town swapped. He doesn't copy it; he builds pages for the five towns where he has real jobs, photos and reviews.
FAQ
Is Screaming Frog free?
Yes, up to 500 URLs per crawl. A licence (£199 or US$279 a year per user, November 2026) adds settings, saved crawls, scheduling and integrations.
Will a competitor know I crawled their site?
Possibly. By default it names itself "Screaming Frog SEO Spider" in every request and obeys their robots.txt; the name can show in their server records.
Can Screaming Frog show a competitor's keywords, traffic or backlinks?
Not alone. It reads Search Console and Analytics only for accounts you can open. Links to your website from other websites. Links from trusted, relevant sites can help your pages rank. More need a licence plus an Ahrefs, Majestic or Moz account.
Different for you?
- You want to know what customers search for: keyword research for a local business.
- You're writing the service pages now: on-page SEO for service pages.
- The rival has a page for every town: city pages for service areas.
- You want links like theirs: backlinks for a local business.
- The rival wins on Maps, not on websites: how to rank higher on Google Maps.
- You'd rather have the comparison and the fixes done for you: SEO & GEO Agent.