Products SEO & GEO Agent Paid Ads Agent Website Agent Conversion Agent
How it works Pricing FAQ Blog Book a call

Screaming Frog for mobile SEO: crawl as a phone and fix

Updated · 8 min read

Short answer With a paid licence, set Screaming Frog's user agent to Googlebot Smartphone, connect PageSpeed Insights under API Access, then crawl. The PageSpeed tab shows speed per page; the Mobile tab flags missing viewport tags and small tap targets. The free version can't do either, so small sites can test key pages in PageSpeed Insights instead.

Want help applying this to your business? Talk it through on a free 30-minute call.

Book a call

How to crawl your site as a phone in Screaming Frog

Change one setting, connect one free Google service, then crawl. Google ranks the phone version of your pages:

"Google uses the mobile version of a site's content, crawled with the smartphone agent, for indexing and ranking." (Google Search Central: Mobile-first indexing best practices)

Steps, from Screaming Frog's user guide and tutorials as of November 2026:

StepWhereWhat to do
1. Switch to a phoneConfiguration > User-AgentPick Googlebot's smartphone preset. The user agent is the name the crawler gives your server when it asks for a page; the same setting decides which robots.txt rules it follows
2. Connect PageSpeed InsightsConfiguration > API Access > PageSpeed InsightsChoose Remote (needs a free API key from Google) or Local (runs Google's Lighthouse test on your computer, no key)
3. Pick phone dataSame window, Metrics tabSelect Mobile, not Desktop. Keep "Mobile Friendly" ticked
4. Find separate phone pagesConfiguration > Spider > CrawlTick Crawl and Store for mobile alternate links. Only matters if your site has an m. address
5. Optional: see pages as a phoneConfiguration > Spider > RenderingChoose JavaScript, with the Googlebot smartphone window size. Screenshots appear in the Rendered Page tab
6. CrawlAddress bar, StartWait for both the crawl bar and the API progress bar to reach 100%

Screaming Frog's mobile tutorial says plainly: "An SEO Spider licence is required." The free version can't change settings or connect PageSpeed Insights (Screaming Frog: SEO Spider user guide, General).

Do you need the paid licence?

Only if you have more pages than you'll test by hand. The licence is £199 a year per user; the US-dollar price shown in November 2026 was $279, and there's no Canadian-dollar price (Screaming Frog: pricing).

Your siteCheapest good route
Under about 20 pages that matterFree: test each in PageSpeed Insights on Mobile, then check the Core Web Vitals report in Search Console
Hundreds of pages, or several sitesLicence: one crawl gives speed and phone checks for every page, in one sheet
A site that needs a loginLicence with Local mode, which "can be used for sites that require authentication" (Screaming Frog: Configuration)

Screaming Frog itself names the gap the licence fills: in PageSpeed Insights "you are limited to one URL at a time", and Search Console shows only good, needs improvement or poor, not each page's numbers (Screaming Frog: How to audit Core Web Vitals).

What changes when you crawl as Googlebot Smartphone?

On most small business sites, very little. If your site uses responsive design (one page that resizes to the screen), Google says "the content and the metadata are the same on the mobile and desktop version" (Google Search Central). Broken links and missing titles show up in either crawl.

The phone setting earns its keep when your server treats phones differently:

  • Dynamic serving: same address, different HTML for phones. Only a phone crawl sees what Google indexes.
  • A separate m. site: Google wants the desktop page as canonical (the main version) and the phone page as its alternate: "The desktop URL is always the canonical, and the mobile version is the alternate of that URL."
  • Phone-only blocks: a noindex tag (do not index) or robots rule served only to phones. Google's rule: "Use the same robots meta tags on the mobile and desktop site."

Google's smartphone crawler identifies itself as Chrome on an Android phone with "Googlebot/2.1" in the name (Google Search Central: Google's common crawlers). If your firewall blocks unknown bots, Screaming Frog's request may be treated differently from Google's real one. Not sure how your site is built? See responsive design and SEO.

Reading the PageSpeed tab: speed per page

This is where most mobile problems sit. The tab holds two kinds of numbers (Screaming Frog: Tabs):

  • CrUX (field data): real Chrome users' visits over the past 28 days, from Google's Chrome User Experience Report. This is what Google's Core Web Vitals measure. Remote mode only; "CrUX data is not available by running Lighthouse locally."
  • Lighthouse (lab data): one simulated load of the page. Useful for finding causes; it is not what real visitors felt.

Sort by the Core Web Vitals Assessment column (Pass or Fail), then by each metric. Google's lines for "good" (Google Search Central: Core Web Vitals):

MetricMeasuresGood
Largest Contentful Paint (LCP)When the main content showsWithin 2.5 seconds
Interaction to Next Paint (INP)How fast a tap gets a responseUnder 200 milliseconds
Cumulative Layout Shift (CLS)How much the page jumps while loadingUnder 0.1

Expect blanks on a small site. Google says a page may lack real-user data "if it has been recently published or has too few samples from real users" (PageSpeed Insights: About). Then use the Lighthouse columns as a rough guide, and fix your most visited pages first. Click a page and open the lower Lighthouse Details tab for the files slowing it down; Reports > PageSpeed exports them all.

One change to watch: Google says it plans "to discontinue including real-world data from the Chrome User Experience Report" in the PageSpeed Insights API (PageSpeed Insights API: Get started). No date is given. When it happens, the CrUX columns from Remote mode may go empty until Screaming Frog adapts.

Reading the Mobile tab: viewport, tap targets, fonts

The Mobile tab scores each check per page; "'100' is a pass, while '0' represents an audit fail" (Screaming Frog: How to audit mobile usability). Its filters:

FilterWhat it meansUsual fix
Viewport Not SetNo viewport tag, so phones show the desktop layout shrunk downAdd a viewport tag set to width=device-width, initial-scale=1 in the page head (Chrome for Developers)
Target SizeButtons or links smaller than 24 by 24 CSS pixels, or too close togetherBigger buttons, more space between menu links
Content Not Sized CorrectlyPage wider or narrower than the screen: sideways scrollingFind the wide element: a table, image or embed with a fixed width
Illegible Font SizeOver 40% of the text is smaller than 12 pixelsRaise the base font size in your theme
Unsupported PluginsFlash, Silverlight or Java appletsReplace with video or plain HTML
Mobile Alternate LinkDesktop pages pointing to a separate phone pageFine, but responsive design is simpler to run

Two filters only work in Local mode. Lighthouse dropped the font-size and content-width checks, so the Remote API no longer returns them (Screaming Frog: How to audit mobile usability).

Bulk exports are under Reports > Mobile.

What to fix first

Fix what stops a customer, on the pages that bring customers:

  1. Viewport Not Set. One line in the page template fixes every page at once.
  2. Content that differs on phones. "Only the content shown on the mobile site is used for indexing" (Google Search Central). Check a few pages on your own phone.
  3. Failing Core Web Vitals on your service and contact pages. See Core Web Vitals on mobile for the fixes.
  4. Tap targets on your call button and menu. A missed tap on "Call now" is a lost lead.

Keep speed in proportion. Google says "there is no single signal" for page experience, though Core Web Vitals "are used by our ranking systems" (Google Search Central: Understanding page experience).

Example

Illustrative cases with invented numbers.

A dental clinic with 35 pages (US dollars)

The clinic needs speed numbers for its 8 treatment pages, not the whole site. The owner runs each through PageSpeed Insights on Mobile, finds two pages with large hero photos failing LCP, and compresses them. Tool cost: $0.

A home services company with 600 pages (Canadian dollars)

City and service pages push the site past the free 500-URL limit. A licence, priced in US dollars, gets PageSpeed data for every page in one crawl. The Mobile tab shows 120 pages with a too-small "Call" button in the same template; one theme fix clears all of them.

A restaurant with an m. site

The phone crawl with mobile alternate links on finds 15 phone pages with no menu text, only images. Google indexes the phone version, so those pages show almost nothing to Google. The fix: put the menu text on the phone pages, or move to one responsive site.

FAQ

Can the free version of Screaming Frog crawl as a phone?

No. Changing the user agent is part of the configuration, which needs a licence. So do PageSpeed Insights data and the Mobile tab (Screaming Frog: General; How to audit mobile usability).

Is the PageSpeed Insights API key free?

Yes. Google creates it from your Google account in a few clicks. The API allows 25,000 queries a day, so in Remote mode Screaming Frog checks at most about 36 pages a minute (Screaming Frog: Configuration).

Why are the CrUX columns empty?

The page has too few real visits in Chrome over 28 days, or you used Local mode. Use the Lighthouse columns, or the site-wide figure PageSpeed Insights shows when a page lacks its own data.

Does Search Console still have a mobile usability report?

No. Google retired it, and the Mobile-Friendly Test, in December 2023. See mobile usability in Search Console for what to check instead.

Different for you?

Keep learning

NextMajestic for local SEO: find local link chances by topicHow a local business uses Majestic's Topical Trust Flow and Clique Hunter to find town and trade sites worth a link, what each plan costs, and the free route.

Ask us one question

One line is enough. We reply by email.

Sources

  1. Screaming Frog: SEO Spider user guide, Configuration
  2. Screaming Frog: SEO Spider user guide, Tabs
  3. Screaming Frog: SEO Spider user guide, General
  4. Screaming Frog: How to audit mobile usability
  5. Screaming Frog: How to audit Core Web Vitals
  6. Screaming Frog: SEO Spider pricing
  7. Google Search Central: Mobile site and mobile-first indexing best practices
  8. Google Search Central: Google's common crawlers
  9. Google Search Central: Understanding Core Web Vitals and Google search results
  10. Google Search Central: Understanding page experience in Google Search results
  11. PageSpeed Insights: About PageSpeed Insights
  12. PageSpeed Insights API: Get started
  13. Google Search Central Blog: The Search Console mobile friendly testing tool (retired)
  14. Chrome for Developers: Lighthouse viewport audit