Products SEO & GEO Agent Paid Ads Agent Website Agent Conversion Agent
How it works Pricing FAQ Blog Book a call

Screaming Frog keyword research: map the keywords on your pages

Updated · 6 min read

Short answer Screaming Frog doesn't find new keywords or show search volumes. It's a crawler: it lists every page on a site with its title and headings. That makes it good for one job, a keyword map: which page targets which search, where two pages chase the same one, and which services have no page. Get volumes from Keyword Planner and real searches from Search Console.

Want help applying this to your business? Talk it through on a free 30-minute call.

Book a call

What Screaming Frog can and can't do for keywords

It reads your pages; it doesn't know what people search. Screaming Frog's SEO Spider is a crawler, a desktop program that follows every link on a site and records each page. It has no keyword database, no search volumes and no rankings.

So split the work across three free tools:

QuestionToolCost
Which keyword does each page target now?Screaming Frog (titles, H1, H2)Free up to 500 URLs
How many people search for a term?Google Keyword PlannerFree, needs a Google Ads account with billing details (Google Ads: Keyword Planner)
Which searches already show my pages?Google Search Console, Performance reportFree

Keyword Planner gives "estimates of the searches they receive and the cost to target them", and you "must enter your billing information and complete your account setup" to use it (Google Ads Help: Use Keyword Planner). Search Console helps you "learn which queries are bringing traffic to your site" (Search Console Help: Performance report).

Free or licence: what you need

For a keyword map on a small site, the free version is enough. It crawls up to 500 URLs and shows page titles and meta data (Screaming Frog: SEO Spider pricing). The licence costs £199 a year and removes the limit (user guide: General).

FeatureFreeLicence
Titles, meta descriptions, H1 and H2 per pageYesYes
Export to a spreadsheetYesYes
Custom Search (find pages that do or don't contain a phrase)NoYes
Search Console data beside each pageNoYes
Semantic similarity (pages about the same thing)NoYes
Save a crawl and reopen itNoYes

Per the pricing page, Custom Search, Search Console integration and semantic similarity are licence-only.

How to build a keyword map, step by step

  1. Crawl. Install the SEO Spider (Windows, macOS or Ubuntu), type your homepage in "Enter URL to spider" and click Start (user guide: General).
  2. Keep only real pages. In the Internal tab, filter to HTML. Skip thank-you, login and tag pages.
  3. Export. Click Export at the top left. The file has each page's address, title, H1 and first H2s. By default the tool collects only "the first two" H1s and H2s on a page (user guide: Tabs), which is plenty for a map.
  4. Add a column: main keyword. For each page, write the one search it should answer, in a customer's words: "water heater repair Tulsa", not "plumbing solutions". Read it off the title and H1. If you can't tell, neither can a search engine reader.
  5. Sort by that column. Duplicates and blanks now sit next to each other.
  6. Add volumes. Paste your main keywords into Keyword Planner and copy its monthly estimates into the sheet. Many local terms show low or no numbers: "Keywords with very low search volumes... aren't discoverable or forecastable" (Google Ads Help: Use Keyword Planner). Low isn't zero.
  7. Add real searches. In Search Console, open the Performance report, filter to one page, then switch to Queries. That's what Google already shows the page for.

What you keep is the map: one row per page, one main keyword each, plus the searches it already wins.

Find two pages chasing one keyword

After sorting, any keyword that appears twice is a conflict, often called keyword cannibalization. Typical small-site cases: a "Drain cleaning" service page and a blog post titled "Drain cleaning in Tulsa", or two near-identical town pages.

Screaming Frog flags some of this for you. Page Titles > Duplicate lists pages with the same title; the tool says identical titles make it "more challenging for users and the search engines to understand one page from another" (user guide: Tabs).

Fix by job, not by rule:

  • Same intent: merge into the stronger page and redirect the other.
  • Different intent: keep both, but give each its own keyword. The service page targets "drain cleaning Tulsa"; the post targets "why does my drain keep clogging".

Not sure if two keywords belong together? Search both and compare the results: SERP overlap.

Find keywords you have no page for

Gaps show up two ways.

  • Services or towns missing from the map. List what you sell and where. Anything with no row is a page you don't have.
  • Searches with no matching page. In Search Console's Queries list, look for terms you rank for by accident, from a page that only mentions them. A strong query on a weak page is a new page waiting to be written.

To see what rivals cover, crawl their site the same way: Screaming Frog competitor analysis.

Check keyword placement without stuffing

The map also shows where a page's keyword is missing. If the title says "Water Heater Repair" and the H1 says "Welcome", the H1 is wasted. Put the main keyword in the title and the H1, in natural words.

Screaming Frog's "Same as h1" filter isn't a problem; the tool says it is "not necessarily an issue". Two more things to ignore:

  • Meta keywords. Google says Google Search "doesn't use the keywords meta tag" (SEO Starter Guide). Screaming Frog's guide says to ignore that tab.
  • Exact wording everywhere. Google says:
"Google's language matching systems are sophisticated and can understand how your page relates to many queries, even if you don't explicitly use the exact terms in them." (Google Search Central: SEO Starter Guide)

It also says repeating words over and over is "against Google's spam policies". One clear keyword per page, plus the related words customers use, is the target: how many keywords per page.

Example

Illustrative cases with invented numbers.

A plumber with 35 pages (US dollars)

The free crawl lists 35 HTML pages. The map shows "drain cleaning" on the service page and on two blog posts. One post repeats the service page; he merges it and redirects. Keyword Planner shows nothing for "hydro jetting Tulsa", but Search Console shows a blog post getting impressions for it, so he writes a service page. Tool cost: $0.

A dental clinic in Ottawa (licence priced in pounds)

The site has 900 URLs, mostly blog posts, so the free crawl stops at 500. The owner buys a licence (£199 a year), runs a Custom Search for "Invisalign" and finds 14 pages using it, but none with it in the title. She makes one page the clear target and links the rest to it.

FAQ

Can Screaming Frog show search volume?

No. It shows what's on your pages, not what people search. Keyword Planner estimates volume; Search Console shows searches you already appear in.

Is the free version enough for keyword research?

For mapping keywords on a site under 500 URLs, yes. Custom Search, Search Console data in the crawl and semantic similarity need the £199-a-year licence.

What is keyword cannibalization?

Two or more of your pages aimed at the same search. Merge them if they do the same job; otherwise give each its own keyword.

Do I need Screaming Frog at all?

Not for 10 to 20 pages; a hand-filled spreadsheet works. The crawler saves time once there are too many pages to check by hand.

Different for you?

Keep learning

NextSemrush keyword research, step by stepHow to do keyword research in Semrush: Keyword Magic Tool match types, the filters worth setting, and saving a short keyword list you can act on.

Ask us one question

One line is enough. We reply by email.

Sources

  1. Screaming Frog: SEO Spider pricing
  2. Screaming Frog: SEO Spider user guide, General
  3. Screaming Frog: SEO Spider user guide, Tabs
  4. Screaming Frog: SEO Spider user guide, Configuration
  5. Google Search Central: SEO Starter Guide
  6. Google Ads: Keyword Planner
  7. Google Ads Help: Use Keyword Planner
  8. Search Console Help: Performance report (Search results)