Screaming Frog keyword research: map the keywords on your pages
Short answer Screaming Frog doesn't find new keywords or show search volumes. It's a crawler: it lists every page on a site with its title and headings. That makes it good for one job, a keyword map: which page targets which search, where two pages chase the same one, and which services have no page. Get volumes from Keyword Planner and real searches from Search Console.
Want help applying this to your business? Talk it through on a free 30-minute call.
Book a callWhat Screaming Frog can and can't do for keywords
It reads your pages; it doesn't know what people search. Screaming Frog's SEO Spider is a crawler, a desktop program that follows every link on a site and records each page. It has no keyword database, no search volumes and no rankings.
So split the work across three free tools:
| Question | Tool | Cost |
|---|---|---|
| Which keyword does each page target now? | Screaming Frog (titles, H1, H2) | Free up to 500 URLs |
| How many people search for a term? | Google Keyword Planner | Free, needs a Google Ads account with billing details (Google Ads: Keyword Planner) |
| Which searches already show my pages? | Google Search Console, Performance report | Free |
Keyword Planner gives "estimates of the searches they receive and the cost to target them", and you "must enter your billing information and complete your account setup" to use it (Google Ads Help: Use Keyword Planner). Search Console helps you "learn which queries are bringing traffic to your site" (Search Console Help: Performance report).
Free or licence: what you need
For a keyword map on a small site, the free version is enough. It crawls up to 500 URLs and shows page titles and meta data (Screaming Frog: SEO Spider pricing). The licence costs £199 a year and removes the limit (user guide: General).
| Feature | Free | Licence |
|---|---|---|
| Titles, meta descriptions, H1 and H2 per page | Yes | Yes |
| Export to a spreadsheet | Yes | Yes |
| Custom Search (find pages that do or don't contain a phrase) | No | Yes |
| Search Console data beside each page | No | Yes |
| Semantic similarity (pages about the same thing) | No | Yes |
| Save a crawl and reopen it | No | Yes |
Per the pricing page, Custom Search, Search Console integration and semantic similarity are licence-only.
How to build a keyword map, step by step
- Crawl. Install the SEO Spider (Windows, macOS or Ubuntu), type your homepage in "Enter URL to spider" and click Start (user guide: General).
- Keep only real pages. In the Internal tab, filter to HTML. Skip thank-you, login and tag pages.
- Export. Click Export at the top left. The file has each page's address, title, H1 and first H2s. By default the tool collects only "the first two" H1s and H2s on a page (user guide: Tabs), which is plenty for a map.
- Add a column: main keyword. For each page, write the one search it should answer, in a customer's words: "water heater repair Tulsa", not "plumbing solutions". Read it off the title and H1. If you can't tell, neither can a search engine reader.
- Sort by that column. Duplicates and blanks now sit next to each other.
- Add volumes. Paste your main keywords into Keyword Planner and copy its monthly estimates into the sheet. Many local terms show low or no numbers: "Keywords with very low search volumes... aren't discoverable or forecastable" (Google Ads Help: Use Keyword Planner). Low isn't zero.
- Add real searches. In Search Console, open the Performance report, filter to one page, then switch to Queries. That's what Google already shows the page for.
What you keep is the map: one row per page, one main keyword each, plus the searches it already wins.
Find two pages chasing one keyword
After sorting, any keyword that appears twice is a conflict, often called keyword cannibalization. Typical small-site cases: a "Drain cleaning" service page and a blog post titled "Drain cleaning in Tulsa", or two near-identical town pages.
Screaming Frog flags some of this for you. Page Titles > Duplicate lists pages with the same title; the tool says identical titles make it "more challenging for users and the search engines to understand one page from another" (user guide: Tabs).
Fix by job, not by rule:
- Same intent: merge into the stronger page and redirect the other.
- Different intent: keep both, but give each its own keyword. The service page targets "drain cleaning Tulsa"; the post targets "why does my drain keep clogging".
Not sure if two keywords belong together? Search both and compare the results: SERP overlap.
Find keywords you have no page for
Gaps show up two ways.
- Services or towns missing from the map. List what you sell and where. Anything with no row is a page you don't have.
- Searches with no matching page. In Search Console's Queries list, look for terms you rank for by accident, from a page that only mentions them. A strong query on a weak page is a new page waiting to be written.
To see what rivals cover, crawl their site the same way: Screaming Frog competitor analysis.
Check keyword placement without stuffing
The map also shows where a page's keyword is missing. If the title says "Water Heater Repair" and the H1 says "Welcome", the H1 is wasted. Put the main keyword in the title and the H1, in natural words.
Screaming Frog's "Same as h1" filter isn't a problem; the tool says it is "not necessarily an issue". Two more things to ignore:
- Meta keywords. Google says Google Search "doesn't use the keywords meta tag" (SEO Starter Guide). Screaming Frog's guide says to ignore that tab.
- Exact wording everywhere. Google says:
"Google's language matching systems are sophisticated and can understand how your page relates to many queries, even if you don't explicitly use the exact terms in them." (Google Search Central: SEO Starter Guide)
It also says repeating words over and over is "against Google's spam policies". One clear keyword per page, plus the related words customers use, is the target: how many keywords per page.
Example
Illustrative cases with invented numbers.
A plumber with 35 pages (US dollars)
The free crawl lists 35 HTML pages. The map shows "drain cleaning" on the service page and on two blog posts. One post repeats the service page; he merges it and redirects. Keyword Planner shows nothing for "hydro jetting Tulsa", but Search Console shows a blog post getting impressions for it, so he writes a service page. Tool cost: $0.
A dental clinic in Ottawa (licence priced in pounds)
The site has 900 URLs, mostly blog posts, so the free crawl stops at 500. The owner buys a licence (£199 a year), runs a Custom Search for "Invisalign" and finds 14 pages using it, but none with it in the title. She makes one page the clear target and links the rest to it.
FAQ
Can Screaming Frog show search volume?
No. It shows what's on your pages, not what people search. Keyword Planner estimates volume; Search Console shows searches you already appear in.
Is the free version enough for keyword research?
For mapping keywords on a site under 500 URLs, yes. Custom Search, Search Console data in the crawl and semantic similarity need the £199-a-year licence.
What is keyword cannibalization?
Two or more of your pages aimed at the same search. Merge them if they do the same job; otherwise give each its own keyword.
Do I need Screaming Frog at all?
Not for 10 to 20 pages; a hand-filled spreadsheet works. The crawler saves time once there are too many pages to check by hand.
Different for you?
- You don't have a keyword list yet: keyword research for a local business.
- You want to start from your own search data: keyword research with Search Console.
- You're choosing a main and supporting keywords for one page: primary and secondary keywords.
- You'd rather have the map built and the pages fixed for you: SEO & GEO Agent.