Products SEO & GEO Agent Paid Ads Agent Website Agent Conversion Agent
How it works Pricing FAQ Blog Book a call

SERP scraper: tools, legality, and build vs buy

Updated · 6 min read

Short answer A SERP scraper is software that runs searches automatically and saves the results as data. Google's spam policies and terms forbid automated queries without its permission, so building your own is the riskiest route. Most owners need none: Search Console shows your own rankings free, and rank trackers or SERP APIs sell the data for cents per search.

Want help applying this to your business? Talk it through on a free 30-minute call.

Book a call

What does a SERP scraper do?

It types searches into a search engine for you, hundreds or thousands at a time, and saves what comes back in a spreadsheet or database. A SERP is the search engine results page: the ads, the map pack, the blue links, People Also Ask boxes and AI answers you see after a search.

What that data can tell a business:

QuestionWhat the scraper records
Where do I rank for my 50 main searches?Your position for each search, each day
Who are my real search competitors?Which sites show up most across your searches
Which searches show ads, maps or AI answers?The features on each page (see SERP features)
What do customers ask?People Also Ask questions and related searches
What do competitors charge or promise?Ad text and product listings, where shown

The value is scale and history. One manual search tells you today's page. A scraper tells you how 200 pages changed over six months.

Google doesn't allow it without permission, and says so in two places. Its spam policies:

"Machine-generated traffic (also called automated traffic) refers to the practice of sending automated queries to Google. This includes scraping results for rank-checking purposes or other types of automated access to Google Search conducted without express permission." (Google Search Central: Spam policies)

The same section says such activity violates its spam policies and Terms of Service. The terms, effective July 30, 2026, ban "using automated means to access content from any of our services in violation of the machine-readable instructions on our web pages". Google's own robots.txt file, those machine-readable instructions, contains the line "Disallow: /search" for all bots.

Whether scraping also breaks a law, not just Google's terms, is still being argued in court. The live case is Google's own:

  • December 19, 2025: Google sued SerpApi, a company that sells Google results as data, "for circumventing security measures protecting others' copyrighted content" (Google Blog).
  • July 2026: the court dismissed Google's original complaint, letting it refile part of it (Search Engine Roundtable).
  • November 9, 2026: a federal judge in Northern California dismissed Google's amended complaint, the second dismissal. Google got 21 days to fix it or the case ends for good, according to SerpApi's general counsel.

So the law is unsettled, and Google's rule is not. What this means for you is a question for a lawyer, not this page. The practical risk for a small business is simpler: Google blocks the traffic, and your scraper stops working.

Do you even need a SERP scraper?

Usually not. Start with the data you can get with permission.

NeedPermitted routeCost
Your own rankings and the searches you show up forGoogle Search Console Performance reportFree
The same data in a spreadsheet or scriptSearch Console API: clicks, impressions and "Average position in search results", up to 25,000 rows per requestFree
A quick look at one searchSearch it yourself, logged out, from the area you serveFree
Daily positions for many keywordsA rank tracker (see how to use a keyword rank checker)Monthly fee
Competitors' rankings, at scaleA paid SEO tool or a SERP APIPer search or monthly

Search Console only covers your own site, and only Google. It tells you what people searched before they found you; it can't tell you who outranks you. That gap is why scrapers exist. Search Console keyword research walks through what it can do.

Build vs buy: which is right?

Buy, if you need the data at all.

Build your ownBuy from a SERP API
SetupA developer, servers, code to read each page layoutSign up, send a search, get structured data back
Keeps working when Google changes its page?Only if someone fixes itThe vendor's problem
Getting blockedYour problem. Avoiding it means disguising traffic, the very thing Google is suing overThe vendor's problem
Google's termsYou are the one sending automated queriesThe vendor sends them; Google still hasn't given permission
Cost at 1,000 searches a monthDeveloper time plus servers$0.60 to $25 (prices below)

Buying does not make the data Google-approved. It moves the work, and the blocking, to someone else. One vendor, SerpApi, sells a "U.S. Legal Shield" with "up to $2 million in coverage for the scraping and parsing of search engine data" on its Production plan ($150 a month) and up, not on cheaper plans (SerpApi pricing).

The official routes are mostly gone. Google says its Custom Search JSON API "is closed to new customers", and existing users "have until January 1, 2027 to transition to an alternative solution." Microsoft retired the Bing Search APIs on August 11, 2025, pointing customers to Grounding with Bing Search inside Azure AI Agents.

What do SERP APIs cost?

Two published price lists, checked November 16, 2026:

VendorFreeEntry pricePer 1,000 searches
SerpApi250 searches a month$25 a month for 1,000$25 at entry, $7.25 on the 100,000 plan ($725)
DataForSEO, Google organicNot listedPay as you go$0.60 (standard queue), $1.20 (priority), $2.00 (live)

DataForSEO's standard queue returns results later; live mode answers right away for more money. A local business tracking 50 searches once a week runs about 200 searches a month, which fits inside SerpApi's free tier.

What the scraped data can't tell you

A scraper sees one version of the page. Google personalizes results:

"To do this, we use information such as your location, past Search history, and Search settings to determine what is most relevant for you in the moment." (Google: How Search works)

For a plumber or a dentist, location matters most. A scraper set to "Chicago" may show a different map pack than a customer two miles from your shop sees. Pick an API that lets you set the city or ZIP code, and treat any position as a range, not a fact.

It also can't tell you what happened after the search: calls, bookings, sales. That lives in your own analytics, and it is the number that pays the bills.

Example

A roofer with 30 keywords

She wants to know if her new service pages are climbing. Search Console's Performance report shows her average position for each search, free. No scraper needed.

A dental group watching five competitors

They want weekly rankings for 40 searches in three cities, including rivals' positions: about 480 searches a month. SerpApi's $25 plan covers 1,000; DataForSEO would cost under $1 a month plus developer time to connect it.

An agency owner thinking of building one

Writing a scraper, renting servers and keeping it unblocked costs far more than $0.60 per 1,000 searches, and puts the automated queries in his name. He buys an API.

FAQ

Is SERP scraping against Google's rules?

Yes. Google's spam policies name "scraping results for rank-checking purposes" as automated traffic it doesn't permit without express permission, and say it violates its Terms of Service.

Will scraping Google hurt my website's rankings?

The policy covers queries sent to Google, not pages on your site. The practical result is blocked requests.

Does Google have an official search API?

Not for new customers. The Custom Search JSON API is closed to new sign-ups, and existing users must move off it by January 1, 2027.

Can I scrape Bing instead?

Microsoft retired its Bing Search APIs on August 11, 2025. Its replacement, Grounding with Bing Search, feeds results to Azure AI agents rather than handing you a results page.

Do rank trackers scrape Google?

Ask the vendor. A cloud tracker sends its queries from its own servers, so the cost and blocking risk are its, not yours.

Different for you?

Keep learning

NextRightBlogger keyword research tool: how to use itHow to find blog keywords with RightBlogger step by step, what each column means, what the free and paid plans cost, and when free Google tools are enough.

Ask us one question

One line is enough. We reply by email.

Sources

  1. Google Search Central: Spam policies for Google web search
  2. Google Terms of Service (effective July 30, 2026)
  3. Google robots.txt
  4. Google Blog: Why we're taking legal action against SerpApi's unlawful scraping (Dec 19, 2025)
  5. SerpApi Blog: Court dismisses Google's amended complaint against SerpApi (Nov 10, 2026)
  6. Search Engine Roundtable: Google Lawsuit Against SerpApi Over Scraping Search Results Dismissed (Jul 22, 2026)
  7. Google for Developers: Custom Search JSON API overview
  8. Microsoft Lifecycle: Bing Search APIs retiring on August 11, 2025
  9. Google for Developers: Search Console API, Search Analytics query
  10. SerpApi: Pricing
  11. DataForSEO: Google Organic SERP API pricing
  12. Google: How Search works, ranking results