JetOctopus website crawl: setup, large-site settings and alerts
Short answer Create a project, click New crawl, enter your homepage, and pick a mode: whole site, sitemap only or a URL list. On big sites, exclude tracking and filter URLs, keep threads low and leave up to 10% of the page limit spare. Link Search Console from your profile menu; schedule crawls and add alerts. Plans start at $349 a month.
Want help applying this to your business? Talk it through on a free 30-minute call.
Book a callJetOctopus website crawl: the setup steps
JetOctopus is a cloud crawler: it walks your site link by link on its own servers, the way Google's crawler does, so your computer does no work. These steps follow JetOctopus's help articles, written 2020–2022; labels in today's app may differ slightly.
| Step | Where | What to set |
|---|---|---|
| 1. Create a project | Dashboard | One site, or one section of it with a clear name |
| 2. Start a crawl | Project > New crawl | Homepage address with https:// |
| 3. Pick a crawl mode | Basic settings | Default (whole site), Only Sitemap, or URL list |
| 4. Set scope and load | Basic settings | Subdomains, robots rules, page limit, threads, JavaScript |
| 5. Exclude junk URLs | Advanced settings | Deny patterns or a custom robots.txt |
| 6. Save and launch | Bottom of the form | "Save as project defaults", then Start crawl |
| 7. Link Search Console | Profile icon > GSC Integration | +Add Google Account |
| 8. Schedule and alert | Project menu | New scheduled crawl, then Create alert |
The crawl settings, one by one
Most can stay as they are. The ones that change your results:
| Setting | Default | Change it when |
|---|---|---|
| Crawl Subdomains | Off: blog.yoursite.com is treated as a separate site | You want subdomains in the same report |
| Crawl mode | Finds every link in your pages' code | You only need sitemap pages, or a list of up to 100,000 URLs |
| Respect robots rules | Not stated | On to see what Google may crawl; off before a redesign to see every page your site generates |
| Execute JavaScript | Not stated | Your text or links only appear after scripts run. Each such page costs 2 pages of your limit |
| Do not crawl URLs with GET parameters | Not stated | Addresses with "?" are filters or tracking, and you know Google can't crawl or index them |
| Pages limit | Not stated | Testing. JetOctopus recommends crawling the whole site |
| Threads | Not stated | Raise only if your hosting can take it. 5 threads act like 5 visitors at once |
| Process sitemaps | Not stated | Tick it to add the URLs in your sitemap to the crawl |
| Process images data | Off | You want image details |
| Cookies | Off: the crawler acts as a new visitor each time | Your server needs them |
Sources: JetOctopus help articles on configuring a crawl, list mode and JavaScript crawls.
For a site with a few hundred pages, enter the homepage, tick Process sitemaps and start. The rest matters when the site is large.
Settings for large websites
Large means tens of thousands of URLs or more. JetOctopus's own advice for big sites:
- Crawl everything if you can. JetOctopus warns a partial crawl can't show how internal links spread value between pages.
- Run a small test crawl first. It shows the URL patterns your site makes: tracking tags, session IDs, filter combinations.
- Exclude only what Google can't reach. Use Deny patterns (one per line; start regular expressions with rx:). JetOctopus's condition: "you must be sure that search engines cannot crawl these URLs." If Google can crawl them, they belong in the report.
- Unsure about your patterns? Use a custom robots.txt. Copy your live robots.txt into Advanced settings, add rules, and test on 2,000–3,000 URLs.
- Leave up to 10% of the page limit spare. It goes to re-crawling pages that returned server errors (5xx) mid-crawl.
- Keep the nofollow option on. Turning it off can hide indexable pages that are linked with rel="nofollow".
- To crawl one folder, use an Allow pattern. Typing the folder as the start address still crawls the whole site.
Do you even have a crawl budget problem? Google says most sites don't:
"If your site doesn't have a large number of pages that change rapidly, or if your pages seem to be crawled the same day that they are published, you don't need to read this guide." (Google Search Central: Optimize your crawl budget)
Google aims that guide at sites with 1 million+ pages that change weekly, or 10,000+ that change daily. See crawl budget.
On JavaScript: turn it on only if your pages need it. Google notes that crawling and reading the HTML "works well for classical websites or server-side rendered pages" (Google Search Central: JavaScript SEO basics). Default render time is 10 seconds; see JavaScript SEO audit.
Link Google Search Console
The link is set once at account level, not per crawl:
- Click your profile icon, top right, then GSC Integration.
- Click +Add Google Account.
- Allow JetOctopus to read your Search Console data. The Google account must already be verified in Search Console.
Plans cap how many Search Console properties you can link: 3 on Growth, 10 on Pro, 20 on Ultra for in-house teams (JetOctopus pricing). If you haven't set Search Console up yet, start with Search Console technical SEO audit, which is free.
Schedule crawls and set alerts
Schedule: in the project menu, click New scheduled crawl. Choose daily, weekly, every two weeks or monthly (or Once, for a later start), the weekday, start date and time zone. Match the frequency to how often your site changes; JetOctopus suggests scheduled crawls to track changes after each release. Every scheduled crawl uses your monthly page allowance.
Alerts: JetOctopus says clients "can set up an unlimited number of alerts for free," in four areas: logs, Search Console, crawl and Core Web Vitals (speed scores). Each alert has three parts:
| Part | What you choose |
|---|---|
| Configure conditions | The data and pages to watch |
| Alert configuration | How often to check and what triggers it; "Test Alert" to try it |
| Send alerts to | Email addresses, Slack, or SMS with country code |
Useful alerts for most sites: a key page stops returning 200 (OK), a key page drops out of Google's index, duplicate titles jump by half after a release. Crawl and Core Web Vitals alerts need a scheduled crawl. Log alerts need your server logs connected directly; manually uploaded logs can't trigger them. For what logs show, see AI crawler server logs.
Does a small business need JetOctopus?
Usually not. Prices in November 2026, per month: Growth $349 (500,000 crawled URLs), Pro $809 (2 million), Ultra $1,559 (5 million), billed annually; quarterly billing costs about a third more. There's a free 7-day trial, opened after a demo, no card needed. Prices are in US dollars or euros; there's no Canadian-dollar price (JetOctopus pricing).
Free route for a site under a few thousand pages:
- Google Search Console: what Google actually crawled and indexed.
- Screaming Frog SEO Spider, free version: crawls up to 500 URLs on your own computer (Screaming Frog pricing).
JetOctopus earns its cost when the site has hundreds of thousands of URLs, or when you need server logs, crawls and Search Console in one place with alerts. For the full audit order, see technical SEO audit.
Example
Illustrative cases with invented numbers.
An online store with 400,000 URLs (US dollars)
A 3,000-URL test crawl shows 70% of addresses are color and size filters, all blocked in robots.txt. The store copies its robots.txt into the custom field, sets the limit to 450,000 to keep 10% spare, and runs on Growth at $349 a month.
A Vancouver home services firm with 80 pages (Canadian dollars)
A demo shows nothing Search Console and a free 500-URL crawl don't already cover. Monthly tool cost stays at $0.
A news site re-checking weekly
A weekly scheduled crawl plus a Slack alert on duplicate titles catches a template bug the morning after a release.
FAQ
How long does a JetOctopus crawl take?
JetOctopus doesn't publish a speed. It depends on site size, threads and JavaScript rendering; more threads finish faster but load your server more.
Does crawling with JetOctopus slow my site?
It can. Each thread acts like one visitor, so start with few threads and add more only if your hosting copes.
Can I crawl just one section?
Yes. Use an Allow pattern for that folder, or paste the addresses into URL list mode (up to 100,000).
Do alerts cost extra?
No, according to JetOctopus. Crawl and speed alerts do run on scheduled crawls, which use your page allowance.
Different for you?
- Your site has under a few thousand pages: Search Console technical SEO audit.
- Google is slow to crawl your new pages: crawl budget.
- Your pages show up empty in the crawl: JavaScript SEO audit.
- You want the full audit method, tool aside: technical SEO audit.
- You'd rather have crawls run and the fixes made for you: SEO & GEO Agent.