Screaming Frog for influencer outreach: what it can do
Short answer Screaming Frog can't find influencers. It crawls websites, and Instagram's robots.txt blocks crawlers it hasn't approved. Find creators in the platforms' own tools or by searching, then paste their blogs into List mode (free up to 500 URLs) to check topic fit, sponsored-post disclosure and contact pages. Pulling contacts in bulk needs the £199-a-year licence.
Want help applying this to your business? Talk it through on a free 30-minute call.
Book a callCan Screaming Frog find influencers?
No. Screaming Frog's SEO Spider is a crawler: a desktop program that opens the web addresses you give it and records what's on each page. It has no database of people, follower counts or audiences. Most influencers live on Instagram, TikTok and YouTube, and those platforms keep crawlers out of the parts that matter.
What it can do is check the websites creators run: a food blogger's site, a local parenting blog, a home-renovation YouTuber's homepage.
| Job | Screaming Frog? | Use instead |
|---|---|---|
| Find creators on Instagram, TikTok or YouTube | No | The platform's own creator tools (below) |
| Show follower counts, engagement or audience location | No | The creator's media kit or the platform's tools |
| Crawl a pasted list of creators' websites | Yes, List mode, free up to 500 URLs | n/a |
| Check what a blog writes about and whether its pages work | Yes, free | n/a |
| See where a sponsored post links and whether those links are marked | Yes, Outlinks tab, free | n/a |
| Flag pages that say "sponsored" or "#ad" across many sites | Yes, Custom Search, licence only | Ctrl+F on each page |
| Pull emails and social profile links into a spreadsheet | Yes, Custom Extraction, licence only | Copy from each contact page |
The licence costs £199 a year per user, and Screaming Frog's pricing page marks Custom Search & Extraction, JavaScript rendering and saved crawls as paid features (pricing page, November 2026).
Why can't it crawl Instagram or TikTok?
The platforms say no, and the tool listens. Screaming Frog's user guide says that "By default the SEO Spider will obey robots.txt protocol" (robots.txt is the file where a site tells crawlers what they may visit). Instagram's robots.txt ends with a rule that blocks every crawler it hasn't named, and opens with this notice:
"Collection of data on Instagram through automated means is prohibited unless you have express written permission from Instagram" (Instagram: robots.txt)
TikTok's robots.txt blocks crawlers from its search pages, including user search, so a crawler can't look creators up there either. Setting the tool to ignore robots.txt or disguise itself goes against what these sites have asked, so don't.
Where do you find influencers instead?
Start with the platforms' own tools, which show audience data Screaming Frog never could:
- Instagram. Meta says its creator marketplace "can help brands easily find relevant creators", and that "Brands can also search for creators, filtering for creator and audience attributes." You join through Meta Business Suite.
- TikTok. TikTok's help center says creator discovery tools sit "directly within your TikTok One account"; some, like AI search, are "available to select customers."
- YouTube. In March 2026 YouTube renamed BrandConnect to YouTube Creator Partnerships, used by advertisers inside Google Ads.
For a local business, the best creators are often found by hand: search Google for `[your city] food blog` or `[your city] mom blog`, check who already tags you, and ask customers which local accounts they follow. Write down each creator's website if they have one. That list is what Screaming Frog checks.
Step 1: list each creator's website pages
For each creator with a site, copy three or four addresses into a text file:
- the homepage
- the about, contact or "work with me" page
- one or two posts that were sponsored or gifted (look for "sponsored", "in partnership with" or "#ad" near the top)
Creators who only have a social account and a link-in-bio page don't need a crawl. Check their profile in the app.
Step 2: crawl the list in List mode
- Download the SEO Spider (Windows, Mac or Ubuntu) and open it.
- Choose Mode > List, click Upload and paste your addresses, each with https://.
- Click Start. Screaming Frog's List mode guide says crawl depth is set to 0, so "only the URLs uploaded will be crawled." Four pages for 40 creators is 160 URLs, inside the free limit.
- If a site returns 403 (forbidden) or 429 (too many requests), it's refusing crawlers. Open it in your browser instead.
Step 3: check fit, upkeep and honesty
Click each page in the top window and read the tabs.
| What you see | Where | What it means |
|---|---|---|
| Page titles and H1 headings about your field or city | Page Titles, H1 tabs | Their readers may be your customers |
| Titles jump between unrelated topics | Page Titles tab | Little real audience; often a site that sells posts |
| Many 404 errors (missing pages) | Response Codes tab | A neglected site; lower priority |
| Sponsored post links marked sponsored or nofollow | Outlinks tab, Follow and Rel columns | The creator follows Google's rules for paid links |
| Sponsored post with exact-keyword links and no disclosure | Outlinks tab, plus reading the post | A risk to you, not just to them (see below) |
Google asks publishers to "Mark links that are advertisements or paid placements (commonly called paid links) with the sponsored value" (Google Search Central: Qualify your outbound links). A creator who does this is used to working with brands properly.
Why disclosure is your problem too
If you pay or gift a creator, the law expects the post to say so. The FTC tells influencers to "Disclose when you have any financial, employment, personal, or family relationship with a brand," and words like "advertisement," "ad," and "sponsored" work. It also puts the brand on the hook:
"Your company is ultimately responsible for what others do on your behalf." (FTC: Endorsement Guides, what people are asking)
Canada is similar. The Competition Bureau's 2018 digest says "advertisers may be liable for representations made through influencers" and tells brands to "Ensure that influencers clearly disclose material connections."
So read the creator's past sponsored posts before you pitch. If none say they were paid or gifted, expect the same with yours.
To keep your notes, click Export on any tab; the List mode guide says it works "on any tab as usual". Add one recent post per creator to mention when you pitch.
Step 4: pull contact details from a list (licence)
With a licence, Custom Extraction (Configuration > Custom > Extraction) fills a column per page. Screaming Frog's scraping tutorial gives `//a[starts-with(@href, 'mailto')]` for email links; swapping in `//a[contains(@href, 'instagram.com')]/@href` collects Instagram profile links the same way. The tutorial notes you "have a licence to access the custom extraction feature."
Without a licence, open each contact page and copy the details. For 20 creators that takes under an hour.
Use the address or form a creator publishes for brand enquiries. Bulk-collecting every address you can find is where spam law starts:
- US. The FTC's CAN-SPAM guide lists "harvesting email addresses" among acts that can bring extra penalties, including imprisonment, and says "The law makes no exception for business-to-business email."
- Canada. CASL needs consent for a commercial email. A published address counts only if it carries no statement refusing unsolicited messages and your message is relevant to the person's business role (CASL, section 10(9)(b)). A "work with me" page invites pitches.
More: cold outreach within spam law and CASL consent.
Do you need Screaming Frog at all?
Usually not. A small business working with five to ten local creators can check each site in a browser in an afternoon. The free crawler helps once you have 30 or more creator sites to compare. The licence pays off only if you vet hundreds of sites and want contact links pulled automatically.
Example
Illustrative cases with invented numbers.
A Calgary bakery (Canadian dollars)
It finds 25 local food creators on Instagram; 12 run blogs. A free crawl of 45 URLs shows three blogs mostly post casino and loan articles. Of the nine left, six disclose past sponsored posts. The bakery offers those six a $50 gift box each, asking for "#ad" on any post.
A Denver HVAC company (US dollars)
A home-renovation blogger wants $400 for a post. The crawl shows her sponsored posts link out with rel="sponsored" and open with "In partnership with". The company books her.
A dental clinic
A parenting blogger has 40,000 Instagram followers but no website. The clinic skips the crawl and checks her past partner posts in the app.
FAQ
Can Screaming Frog scrape Instagram followers or emails?
No. Instagram's robots.txt blocks all unnamed crawlers, and Screaming Frog obeys robots.txt by default. Use Instagram's creator marketplace or the creator's own contact details.
Is Screaming Frog free for influencer research?
The free version crawls up to 500 URLs and shows titles, headings, response codes and outlinks. Custom Search and Custom Extraction need the £199-a-year licence.
Can I measure an influencer's audience with Screaming Frog?
No. It sees website pages, not followers or engagement. Ask for the creator's platform insights or use the platform's marketplace.
Should a sponsored blog post link to me with a normal link?
No. Google asks for paid links to be marked sponsored. The post can still send you visitors and customers; track them with tagged links (influencer tracking).
Different for you?
- You're writing what the creator should post: influencer brief.
- You want to know which creator brought sales: Google Analytics for influencer outreach.
- You want to crawl a rival's site instead: Screaming Frog competitor analysis.
- You want links from sites, not posts from creators: link prospecting.
- You'd rather have creator ads run for you: Paid Ads Agent.