Products SEO & GEO Agent Paid Ads Agent Website Agent Conversion Agent
How it works Pricing FAQ Blog Book a call

Crawling and indexing: how Google finds and stores your pages

Updated · 8 min read

Short answer Google works in order. It discovers a page's address through a link or a sitemap, crawls it by downloading the page, renders it by running its code like a browser, then decides whether to index it, meaning store it in its database. Only indexed pages can show in search. A page can stop at any stage.

Want help applying this to your business? Talk it through on a free 30-minute call.

Book a call

How does Google find and store a page?

In four steps, and each one must succeed before the next starts. Google's guide to how Search works says it plainly:

"Google Search works in three stages, and not all pages make it through each stage" (Google Search Central: How Google Search works)

Those three are crawling, indexing and serving results. Google's own guide splits two more steps out of crawling: finding the address first, and rendering the page while crawling. For an owner, it reads best as four steps before a page can be shown.

StepWhat Google doesWhat stops it
1. DiscoverLearns the page's web address (URL) existsNo link points to it and it isn't in a sitemap
2. CrawlGooglebot, Google's robot, downloads the pagerobots.txt blocks it, a login wall, a server error
3. RenderRuns the page's code in a Chrome browser to see what a visitor seesText or links that appear only after clicks, or code that fails
4. IndexReads the page, picks one version of duplicates, stores itA noindex tag, thin or low-quality content, a near copy of another page

Only after step 4 can a page compete in step 5, serving: showing up for a search. Ranking is a separate question; this guide is about getting into the race.

How does Google discover a new page?

Through links and sitemaps. Google keeps a list of known addresses and adds to it in three ways: pages it has visited before, links it finds on pages it already knows, and lists of pages you send it, called sitemaps (Google Search Central: How Google Search works).

Links from your own pages count, not only links from other websites. A new service page linked from your home page menu is easy to find. A page that nothing links to, and that isn't in your sitemap, may never be found. Links from other sites help most when your site is new.

Do you need a sitemap? Google says you might not if your site is small, which it defines as "about 500 pages or fewer", and every important page can be reached by clicking from the home page. It suggests one if your site is large, new with few links from other sites, or heavy on videos and images (Google Search Central: Learn about sitemaps). Most website builders make one for you. A sitemap helps but promises nothing:

"A sitemap helps search engines discover URLs on your site, but it doesn't guarantee that all the items in your sitemap will be crawled and indexed." (Google Search Central: Learn about sitemaps)

What happens when Google crawls a page?

Googlebot downloads it, if it may and if it can. Google says a program decides "which sites to crawl, how often, and how many pages to fetch from each site", and that Googlebot "doesn't crawl all the pages it discovered" (Google Search Central: How Google Search works). It also slows down on its own when your server struggles: server errors in the 500 range tell it to back off.

Common reasons a crawl fails, per Google: server problems, network issues, and robots.txt rules. robots.txt is a small text file at the root of your site that tells robots which addresses to skip.

Paying doesn't help. Google says it doesn't take payment to crawl a site more often.

Should a small business worry about "crawl budget", the limit on how much Google crawls? Usually not. Google wrote its crawl budget guide for sites with over 1 million pages that change weekly, or over 10,000 pages that change daily. For everyone else it says keeping your sitemap up to date and checking the Page indexing report is enough (Google Search Central: Crawl budget management for large sites). A fast server still helps visitors, but a 30-page site won't run out of crawl.

What is rendering, and why does JavaScript matter?

Rendering is Google running your page's code the way a browser does. Many sites build their text with JavaScript, code that fills the page after it loads. Google renders with a recent version of Chrome so it can see that text; without rendering, it "might not see that content" (Google Search Central: How Google Search works).

Rendering can lag. Google's JavaScript guide says pages wait in a rendering queue, sometimes for seconds, sometimes longer, and that it can't easily tell you whether a page is waiting to be crawled or rendered. It still recommends sending finished HTML from the server, because "not all bots can run JavaScript" (Google Search Central: Understand JavaScript SEO basics).

Two checks matter here:

  • Links must be real links. Google says it can generally crawl a link only if it's a standard HTML link (an a element with an href address). A button that moves visitors with a script may be a dead end for Googlebot (Google Search Central: Link best practices).
  • The phone version is the one that counts. Google uses the mobile version of a site's content for indexing and ranking, called mobile-first indexing (Google Search Central: Mobile site and mobile-first indexing). Text you hide on phones may be text Google never reads.

Website builders like Wix, Squarespace and Shopify use standard links and phone layouts by default. Custom-built sites are where rendering problems hide. See website platforms and SEO.

What does indexing mean?

Indexing means Google read the page, understood it, and stored it in the Google index, a huge database it searches when someone types a query. Google looks at the text, the page title, image descriptions (alt text), images and videos.

It also groups near-identical pages and keeps one as the main version, called the canonical. The others are alternates. Google then notes signals about the main version, such as its language, the country it serves, and how usable it is (Google Search Central: How Google Search works).

Indexing is not automatic:

"Indexing isn't guaranteed; not every page that Google processes will be indexed." (Google Search Central: How Google Search works)

Google names three common reasons: low-quality content, a robots meta tag that says not to index, and site design that makes indexing hard.

Duplicate pages don't earn a penalty. Google says some duplicate content on a site "is normal and it's not a violation of Google's spam policies" (Google Search Central: What is canonicalization). The cost is that Google, not you, picks which version to show. Tell it your choice with a canonical tag or a redirect.

How do I keep a page out of Google: robots.txt or noindex?

Use noindex. Blocking a page in robots.txt stops crawling, not indexing:

"it is not a mechanism for keeping a web page out of Google" (Google Search Central: Introduction to robots.txt)

A blocked page can still show in results, with no description under it. And if you block a page in robots.txt and add noindex, Google never sees the noindex, because it can't crawl the page to read it (Google Search Central: Block Search indexing with noindex). For AI crawlers, see blocking AI crawlers with robots.txt.

How do I check each stage on my own site?

Use Google Search Console, Google's free site report. Here is where each stage shows up.

StageCheckWhere
DiscoverYour sitemap is submitted and readSearch Console, Sitemaps report
CrawlNo robots.txt rule blocks the page; last crawl dateSearch Console, URL Inspection
RenderThe tested page shows your text and linksURL Inspection, live test
IndexThe page reads "indexed" and Google chose your canonicalURL Inspection; Page indexing report

After you add or change a page, you can ask for a crawl in URL Inspection. Google says crawling "can take anywhere from a few days to a few weeks", that there's a quota on requests, and that asking again won't make it faster (Google Search Central: Ask Google to recrawl your URLs). For a page labeled "Crawled" or "Discovered – currently not indexed", see crawled or discovered, currently not indexed. To learn the reports, see Search Console for a small business.

Example

Illustrative cases.

A new landscaping site with 12 pages

The home page is indexed after two weeks; the five service pages aren't found. The menu uses script buttons, not real links, so Googlebot can't follow them. The builder's standard menu fixes it. The owner submits the sitemap and requests indexing for the five pages.

A dental practice with duplicate pages

The site has /cleaning and /teeth-cleaning with the same text. Search Console shows Google picked /cleaning. The office redirects the other one to it. No penalty was involved; the redirect just removes the guesswork.

A contractor who hid a staging site

A developer blocked the test copy of the site in robots.txt, and it still shows in Google with no description. The fix is a password on the test site, which keeps both people and Google out.

FAQ

Is crawling the same as indexing?

No. Crawling is Google downloading a page. Indexing is Google deciding to store it. A page can be crawled and still not indexed.

Does a page need backlinks to get indexed?

No. Google finds pages through your own links and your sitemap too. Links from other sites help a brand-new site get found sooner.

Can I pay Google to index my site faster?

No. Google says it doesn't accept payment to crawl a site more often or to rank it higher.

Is Google's Mobile-Friendly Test still available?

No. Google retired the Mobile-Friendly Test and the Mobile Usability report on December 1, 2023 (Google Search Central Blog). URL Inspection's live test still shows how Googlebot sees a page.

Different for you?

Keep learning

NextIndexing in SEO: What It Means and How to Check YoursWhat indexing means in plain words, how it differs from crawling and ranking, and a two-minute check to see whether Google has your pages.

Do it yourself

Crawlability checklist: make every page reachable by GoogleA crawlability checklist for a small business site: find orphan pages, robots.txt blocks, broken links, redirect chains and server errors, with free tools.

Ask us one question

One line is enough. We reply by email.

Sources

  1. Google Search Central: In-depth guide to how Google Search works
  2. Google Search Central: Learn about sitemaps
  3. Google Search Central: Introduction to robots.txt
  4. Google Search Central: Block Search indexing with noindex
  5. Google Search Central: Crawl budget management for large sites
  6. Google Search Central: Understand JavaScript SEO basics
  7. Google Search Central: Link best practices for Google
  8. Google Search Central: Mobile site and mobile-first indexing best practices
  9. Google Search Central: What is canonicalization
  10. Google Search Central: Ask Google to recrawl your URLs
  11. Google Search Central Blog: The Search Console mobile friendly testing tool (retired)