Products SEO & GEO Agent Paid Ads Agent Website Agent Conversion Agent
How it works Pricing FAQ Blog Book a call

JetOctopus website crawl: setup, large-site settings and alerts

Updated · 7 min read

Short answer Create a project, click New crawl, enter your homepage, and pick a mode: whole site, sitemap only or a URL list. On big sites, exclude tracking and filter URLs, keep threads low and leave up to 10% of the page limit spare. Link Search Console from your profile menu; schedule crawls and add alerts. Plans start at $349 a month.

Want help applying this to your business? Talk it through on a free 30-minute call.

Book a call

JetOctopus website crawl: the setup steps

JetOctopus is a cloud crawler: it walks your site link by link on its own servers, the way Google's crawler does, so your computer does no work. These steps follow JetOctopus's help articles, written 2020–2022; labels in today's app may differ slightly.

StepWhereWhat to set
1. Create a projectDashboardOne site, or one section of it with a clear name
2. Start a crawlProject > New crawlHomepage address with https://
3. Pick a crawl modeBasic settingsDefault (whole site), Only Sitemap, or URL list
4. Set scope and loadBasic settingsSubdomains, robots rules, page limit, threads, JavaScript
5. Exclude junk URLsAdvanced settingsDeny patterns or a custom robots.txt
6. Save and launchBottom of the form"Save as project defaults", then Start crawl
7. Link Search ConsoleProfile icon > GSC Integration+Add Google Account
8. Schedule and alertProject menuNew scheduled crawl, then Create alert

The crawl settings, one by one

Most can stay as they are. The ones that change your results:

SettingDefaultChange it when
Crawl SubdomainsOff: blog.yoursite.com is treated as a separate siteYou want subdomains in the same report
Crawl modeFinds every link in your pages' codeYou only need sitemap pages, or a list of up to 100,000 URLs
Respect robots rulesNot statedOn to see what Google may crawl; off before a redesign to see every page your site generates
Execute JavaScriptNot statedYour text or links only appear after scripts run. Each such page costs 2 pages of your limit
Do not crawl URLs with GET parametersNot statedAddresses with "?" are filters or tracking, and you know Google can't crawl or index them
Pages limitNot statedTesting. JetOctopus recommends crawling the whole site
ThreadsNot statedRaise only if your hosting can take it. 5 threads act like 5 visitors at once
Process sitemapsNot statedTick it to add the URLs in your sitemap to the crawl
Process images dataOffYou want image details
CookiesOff: the crawler acts as a new visitor each timeYour server needs them

Sources: JetOctopus help articles on configuring a crawl, list mode and JavaScript crawls.

For a site with a few hundred pages, enter the homepage, tick Process sitemaps and start. The rest matters when the site is large.

Settings for large websites

Large means tens of thousands of URLs or more. JetOctopus's own advice for big sites:

  1. Crawl everything if you can. JetOctopus warns a partial crawl can't show how internal links spread value between pages.
  2. Run a small test crawl first. It shows the URL patterns your site makes: tracking tags, session IDs, filter combinations.
  3. Exclude only what Google can't reach. Use Deny patterns (one per line; start regular expressions with rx:). JetOctopus's condition: "you must be sure that search engines cannot crawl these URLs." If Google can crawl them, they belong in the report.
  4. Unsure about your patterns? Use a custom robots.txt. Copy your live robots.txt into Advanced settings, add rules, and test on 2,000–3,000 URLs.
  5. Leave up to 10% of the page limit spare. It goes to re-crawling pages that returned server errors (5xx) mid-crawl.
  6. Keep the nofollow option on. Turning it off can hide indexable pages that are linked with rel="nofollow".
  7. To crawl one folder, use an Allow pattern. Typing the folder as the start address still crawls the whole site.

Do you even have a crawl budget problem? Google says most sites don't:

"If your site doesn't have a large number of pages that change rapidly, or if your pages seem to be crawled the same day that they are published, you don't need to read this guide." (Google Search Central: Optimize your crawl budget)

Google aims that guide at sites with 1 million+ pages that change weekly, or 10,000+ that change daily. See crawl budget.

On JavaScript: turn it on only if your pages need it. Google notes that crawling and reading the HTML "works well for classical websites or server-side rendered pages" (Google Search Central: JavaScript SEO basics). Default render time is 10 seconds; see JavaScript SEO audit.

The link is set once at account level, not per crawl:

  1. Click your profile icon, top right, then GSC Integration.
  2. Click +Add Google Account.
  3. Allow JetOctopus to read your Search Console data. The Google account must already be verified in Search Console.

Plans cap how many Search Console properties you can link: 3 on Growth, 10 on Pro, 20 on Ultra for in-house teams (JetOctopus pricing). If you haven't set Search Console up yet, start with Search Console technical SEO audit, which is free.

Schedule crawls and set alerts

Schedule: in the project menu, click New scheduled crawl. Choose daily, weekly, every two weeks or monthly (or Once, for a later start), the weekday, start date and time zone. Match the frequency to how often your site changes; JetOctopus suggests scheduled crawls to track changes after each release. Every scheduled crawl uses your monthly page allowance.

Alerts: JetOctopus says clients "can set up an unlimited number of alerts for free," in four areas: logs, Search Console, crawl and Core Web Vitals (speed scores). Each alert has three parts:

PartWhat you choose
Configure conditionsThe data and pages to watch
Alert configurationHow often to check and what triggers it; "Test Alert" to try it
Send alerts toEmail addresses, Slack, or SMS with country code

Useful alerts for most sites: a key page stops returning 200 (OK), a key page drops out of Google's index, duplicate titles jump by half after a release. Crawl and Core Web Vitals alerts need a scheduled crawl. Log alerts need your server logs connected directly; manually uploaded logs can't trigger them. For what logs show, see AI crawler server logs.

Does a small business need JetOctopus?

Usually not. Prices in November 2026, per month: Growth $349 (500,000 crawled URLs), Pro $809 (2 million), Ultra $1,559 (5 million), billed annually; quarterly billing costs about a third more. There's a free 7-day trial, opened after a demo, no card needed. Prices are in US dollars or euros; there's no Canadian-dollar price (JetOctopus pricing).

Free route for a site under a few thousand pages:

  • Google Search Console: what Google actually crawled and indexed.
  • Screaming Frog SEO Spider, free version: crawls up to 500 URLs on your own computer (Screaming Frog pricing).

JetOctopus earns its cost when the site has hundreds of thousands of URLs, or when you need server logs, crawls and Search Console in one place with alerts. For the full audit order, see technical SEO audit.

Example

Illustrative cases with invented numbers.

An online store with 400,000 URLs (US dollars)

A 3,000-URL test crawl shows 70% of addresses are color and size filters, all blocked in robots.txt. The store copies its robots.txt into the custom field, sets the limit to 450,000 to keep 10% spare, and runs on Growth at $349 a month.

A Vancouver home services firm with 80 pages (Canadian dollars)

A demo shows nothing Search Console and a free 500-URL crawl don't already cover. Monthly tool cost stays at $0.

A news site re-checking weekly

A weekly scheduled crawl plus a Slack alert on duplicate titles catches a template bug the morning after a release.

FAQ

How long does a JetOctopus crawl take?

JetOctopus doesn't publish a speed. It depends on site size, threads and JavaScript rendering; more threads finish faster but load your server more.

Does crawling with JetOctopus slow my site?

It can. Each thread acts like one visitor, so start with few threads and add more only if your hosting copes.

Can I crawl just one section?

Yes. Use an Allow pattern for that folder, or paste the addresses into URL list mode (up to 100,000).

Do alerts cost extra?

No, according to JetOctopus. Crawl and speed alerts do run on scheduled crawls, which use your page allowance.

Different for you?

Keep learning

NextJS vs non-JS rendered pages: compare them with JetOctopusCompare a page with and without JavaScript in JetOctopus's free checker, read each row, and find content Google or AI crawlers like ChatGPT's can miss.

Ask us one question

One line is enough. We reply by email.

Sources

  1. JetOctopus: How to configure a crawl of your website (May 2022)
  2. JetOctopus: How to create a new crawl (2020)
  3. JetOctopus: How to crawl large websites (April 2022)
  4. JetOctopus: How to use the list mode (2022)
  5. JetOctopus: How to configure crawl for JavaScript websites (June 2022)
  6. JetOctopus: How to schedule a crawl (May 2022)
  7. JetOctopus: How to connect Google Search Console with JetOctopus
  8. JetOctopus: Guide to creating alerts (July 2022)
  9. JetOctopus: Pricing
  10. Google Search Central: Optimize your crawl budget
  11. Google Search Central: Understand the JavaScript SEO basics
  12. Screaming Frog: SEO Spider pricing