Products SEO & GEO Agent Paid Ads Agent Website Agent Conversion Agent
How it works Pricing FAQ Blog Book a call

Screaming Frog for influencer outreach: what it can do

Updated · 7 min read

Short answer Screaming Frog can't find influencers. It crawls websites, and Instagram's robots.txt blocks crawlers it hasn't approved. Find creators in the platforms' own tools or by searching, then paste their blogs into List mode (free up to 500 URLs) to check topic fit, sponsored-post disclosure and contact pages. Pulling contacts in bulk needs the £199-a-year licence.

Want help applying this to your business? Talk it through on a free 30-minute call.

Book a call

Can Screaming Frog find influencers?

No. Screaming Frog's SEO Spider is a crawler: a desktop program that opens the web addresses you give it and records what's on each page. It has no database of people, follower counts or audiences. Most influencers live on Instagram, TikTok and YouTube, and those platforms keep crawlers out of the parts that matter.

What it can do is check the websites creators run: a food blogger's site, a local parenting blog, a home-renovation YouTuber's homepage.

JobScreaming Frog?Use instead
Find creators on Instagram, TikTok or YouTubeNoThe platform's own creator tools (below)
Show follower counts, engagement or audience locationNoThe creator's media kit or the platform's tools
Crawl a pasted list of creators' websitesYes, List mode, free up to 500 URLsn/a
Check what a blog writes about and whether its pages workYes, freen/a
See where a sponsored post links and whether those links are markedYes, Outlinks tab, freen/a
Flag pages that say "sponsored" or "#ad" across many sitesYes, Custom Search, licence onlyCtrl+F on each page
Pull emails and social profile links into a spreadsheetYes, Custom Extraction, licence onlyCopy from each contact page

The licence costs £199 a year per user, and Screaming Frog's pricing page marks Custom Search & Extraction, JavaScript rendering and saved crawls as paid features (pricing page, November 2026).

Why can't it crawl Instagram or TikTok?

The platforms say no, and the tool listens. Screaming Frog's user guide says that "By default the SEO Spider will obey robots.txt protocol" (robots.txt is the file where a site tells crawlers what they may visit). Instagram's robots.txt ends with a rule that blocks every crawler it hasn't named, and opens with this notice:

"Collection of data on Instagram through automated means is prohibited unless you have express written permission from Instagram" (Instagram: robots.txt)

TikTok's robots.txt blocks crawlers from its search pages, including user search, so a crawler can't look creators up there either. Setting the tool to ignore robots.txt or disguise itself goes against what these sites have asked, so don't.

Where do you find influencers instead?

Start with the platforms' own tools, which show audience data Screaming Frog never could:

  • Instagram. Meta says its creator marketplace "can help brands easily find relevant creators", and that "Brands can also search for creators, filtering for creator and audience attributes." You join through Meta Business Suite.
  • TikTok. TikTok's help center says creator discovery tools sit "directly within your TikTok One account"; some, like AI search, are "available to select customers."
  • YouTube. In March 2026 YouTube renamed BrandConnect to YouTube Creator Partnerships, used by advertisers inside Google Ads.

For a local business, the best creators are often found by hand: search Google for `[your city] food blog` or `[your city] mom blog`, check who already tags you, and ask customers which local accounts they follow. Write down each creator's website if they have one. That list is what Screaming Frog checks.

Step 1: list each creator's website pages

For each creator with a site, copy three or four addresses into a text file:

  • the homepage
  • the about, contact or "work with me" page
  • one or two posts that were sponsored or gifted (look for "sponsored", "in partnership with" or "#ad" near the top)

Creators who only have a social account and a link-in-bio page don't need a crawl. Check their profile in the app.

Step 2: crawl the list in List mode

  1. Download the SEO Spider (Windows, Mac or Ubuntu) and open it.
  2. Choose Mode > List, click Upload and paste your addresses, each with https://.
  3. Click Start. Screaming Frog's List mode guide says crawl depth is set to 0, so "only the URLs uploaded will be crawled." Four pages for 40 creators is 160 URLs, inside the free limit.
  4. If a site returns 403 (forbidden) or 429 (too many requests), it's refusing crawlers. Open it in your browser instead.

Step 3: check fit, upkeep and honesty

Click each page in the top window and read the tabs.

What you seeWhereWhat it means
Page titles and H1 headings about your field or cityPage Titles, H1 tabsTheir readers may be your customers
Titles jump between unrelated topicsPage Titles tabLittle real audience; often a site that sells posts
Many 404 errors (missing pages)Response Codes tabA neglected site; lower priority
Sponsored post links marked sponsored or nofollowOutlinks tab, Follow and Rel columnsThe creator follows Google's rules for paid links
Sponsored post with exact-keyword links and no disclosureOutlinks tab, plus reading the postA risk to you, not just to them (see below)

Google asks publishers to "Mark links that are advertisements or paid placements (commonly called paid links) with the sponsored value" (Google Search Central: Qualify your outbound links). A creator who does this is used to working with brands properly.

Why disclosure is your problem too

If you pay or gift a creator, the law expects the post to say so. The FTC tells influencers to "Disclose when you have any financial, employment, personal, or family relationship with a brand," and words like "advertisement," "ad," and "sponsored" work. It also puts the brand on the hook:

"Your company is ultimately responsible for what others do on your behalf." (FTC: Endorsement Guides, what people are asking)

Canada is similar. The Competition Bureau's 2018 digest says "advertisers may be liable for representations made through influencers" and tells brands to "Ensure that influencers clearly disclose material connections."

So read the creator's past sponsored posts before you pitch. If none say they were paid or gifted, expect the same with yours.

To keep your notes, click Export on any tab; the List mode guide says it works "on any tab as usual". Add one recent post per creator to mention when you pitch.

Step 4: pull contact details from a list (licence)

With a licence, Custom Extraction (Configuration > Custom > Extraction) fills a column per page. Screaming Frog's scraping tutorial gives `//a[starts-with(@href, 'mailto')]` for email links; swapping in `//a[contains(@href, 'instagram.com')]/@href` collects Instagram profile links the same way. The tutorial notes you "have a licence to access the custom extraction feature."

Without a licence, open each contact page and copy the details. For 20 creators that takes under an hour.

Use the address or form a creator publishes for brand enquiries. Bulk-collecting every address you can find is where spam law starts:

  • US. The FTC's CAN-SPAM guide lists "harvesting email addresses" among acts that can bring extra penalties, including imprisonment, and says "The law makes no exception for business-to-business email."
  • Canada. CASL needs consent for a commercial email. A published address counts only if it carries no statement refusing unsolicited messages and your message is relevant to the person's business role (CASL, section 10(9)(b)). A "work with me" page invites pitches.

More: cold outreach within spam law and CASL consent.

Do you need Screaming Frog at all?

Usually not. A small business working with five to ten local creators can check each site in a browser in an afternoon. The free crawler helps once you have 30 or more creator sites to compare. The licence pays off only if you vet hundreds of sites and want contact links pulled automatically.

Example

Illustrative cases with invented numbers.

A Calgary bakery (Canadian dollars)

It finds 25 local food creators on Instagram; 12 run blogs. A free crawl of 45 URLs shows three blogs mostly post casino and loan articles. Of the nine left, six disclose past sponsored posts. The bakery offers those six a $50 gift box each, asking for "#ad" on any post.

A Denver HVAC company (US dollars)

A home-renovation blogger wants $400 for a post. The crawl shows her sponsored posts link out with rel="sponsored" and open with "In partnership with". The company books her.

A dental clinic

A parenting blogger has 40,000 Instagram followers but no website. The clinic skips the crawl and checks her past partner posts in the app.

FAQ

Can Screaming Frog scrape Instagram followers or emails?

No. Instagram's robots.txt blocks all unnamed crawlers, and Screaming Frog obeys robots.txt by default. Use Instagram's creator marketplace or the creator's own contact details.

Is Screaming Frog free for influencer research?

The free version crawls up to 500 URLs and shows titles, headings, response codes and outlinks. Custom Search and Custom Extraction need the £199-a-year licence.

Can I measure an influencer's audience with Screaming Frog?

No. It sees website pages, not followers or engagement. Ask for the creator's platform insights or use the platform's marketplace.

Should a sponsored blog post link to me with a normal link?

No. Google asks for paid links to be marked sponsored. The post can still send you visitors and customers; track them with tagged links (influencer tracking).

Different for you?

Keep learning

NextContent Explorer for guest posts: find sites and topicsHow to use Ahrefs Content Explorer to find guest post sites, unlinked mentions and low-competition topics, which plan has it, and the free route.

Ask us one question

One line is enough. We reply by email.

Sources

  1. Screaming Frog: SEO Spider pricing
  2. Screaming Frog: SEO Spider user guide, Configuration
  3. Screaming Frog: SEO Spider user guide, Tabs
  4. Screaming Frog: How to use List mode
  5. Screaming Frog: Web scraping and data extraction tutorial
  6. Instagram: robots.txt
  7. TikTok: robots.txt
  8. Meta for Business: Instagram's creator marketplace expansion (2024)
  9. TikTok Business Help Center: Find creators with the creator leaderboard
  10. YouTube Official Blog: brand and creator partnerships (March 2026)
  11. FTC: Disclosures 101 for Social Media Influencers
  12. FTC: Endorsement Guides, what people are asking
  13. Competition Bureau Canada: Deceptive Marketing Practices Digest, Volume 4 (2018)
  14. Google Search Central: Qualify your outbound links to Google
  15. FTC: CAN-SPAM Act, a compliance guide for business (archived 2026-10-31)
  16. Justice Laws Website: Canada's Anti-Spam Legislation, section 10