Products SEO & GEO Agent Paid Ads Agent Website Agent Conversion Agent
How it works Pricing FAQ Blog Book a call

Automatic indexing: what you can automate to get pages found

Updated · 5 min read

Short answer You can automate telling search engines about pages, not getting them indexed. Google decides on its own and says it doesn't guarantee indexing. Automate three things: a sitemap your website builder updates itself, IndexNow for Bing and some other engines, and links to every new page. Skip tools that sell instant Google indexing.

Want help applying this to your business? Talk it through on a free 30-minute call.

Book a call

Can you make Google index pages automatically?

No. You can automate the announcement, not the result. Indexing means a search engine has read your page and stored it so it can appear in results. Google makes that call itself:

"Google doesn't guarantee that it will crawl, index, or serve your page, even if your page follows the Google Search Essentials." (Google Search Central: How Google Search works)

So "automatic indexing" for a business site means one thing: every new or changed page gets announced to search engines without you remembering to do it. That part is easy and free.

"Automatic indexing" also has an older meaning: software that tags documents with keywords, as in library catalogs. Tagging your own pages with keywords doesn't get them into Google. Google says the meta keywords tag "is not used by Google Search, and it has no effect on indexing and ranking at all" (Google Search Central: Meta tags Google supports).

What you can automate, and what each one does

StepWhat runs on its ownWhich enginesSetup
XML sitemapYour builder lists every page and updates the file when you publishGoogle, Bing and othersUsually none. Google says WordPress, Wix and Blogger sites likely have one already
Sitemap line in robots.txtEngines find the sitemap without you submitting it againGoogle, BingOne line: `Sitemap: https://example.com/sitemap.xml`
Honest lastmod datesThe sitemap shows which pages really changedGoogle, BingCheck your platform doesn't stamp every page with today's date
IndexNowEach publish, edit or delete is sent to participating enginesBing, Yandex, Seznam, Naver, Yep, Internet Archive, Amazonbot. Not GoogleA switch in many platforms or plugins
Internal linksA new page shows up in your menu, blog list or service page, where crawlers already goAllSet your theme to list new posts or services

Google's guide to how Search works says it discovers new pages through links from pages it already knows, such as a category page linking to a new post. That last row is the one people forget, and it works on every engine.

For the setup details of each row, see sitemap best practices and submit a sitemap to Google and Bing.

What you can't automate for Google

Three things look automatable and aren't.

Request indexing. Search Console's URL Inspection tool has a Request indexing button, and only a person clicks it. Google's page on recrawling says there's a quota, that asking again "won't get it crawled any faster", and that "If you have large numbers of URLs, submit a sitemap" (Google Search Central: Ask Google to recrawl your URLs).

The URL Inspection API. Some tools use this to check pages in bulk. It only reports. Google describes it as a way to "View the indexed, or indexable, status of the provided URL" (Search Console API reference). It sends no indexing request.

Sitemap pings. Google announced in June 2023 that its sitemap ping address would stop working in six months; the post now says the deprecation is complete (Google Search Central Blog). Plugins that still ping Google do nothing.

Do "instant indexing" tools work?

Not for a normal business page on Google. Most of these tools and plugins push your pages through Google's Indexing API. Google limits that API to two page types:

"The Indexing API can only be used to crawl pages with either JobPosting or BroadcastEvent embedded in a VideoObject." (Google Search Central: Indexing API quickstart)

That means job ads and livestream videos. A roofing page or a menu isn't either. Google also watches for tools that spread requests across many accounts to get around the daily limit:

"All submissions through the Indexing API undergo rigorous spam detection. Any attempts to abuse the Indexing API, including the use of multiple accounts or other means to exceed usage quotas, may result in access being revoked." (Google Search Central: Indexing API quickstart)

How to spot a tool that overpromises:

  • It says "instant", "guaranteed" or "forced" Google indexing. Google guarantees none of these, to anyone.
  • It asks you to create Google Cloud service accounts or upload a JSON key for the Indexing API, and your pages aren't job ads or livestreams.
  • It charges per URL for something a sitemap and IndexNow do for free.

IndexNow itself doesn't promise indexing either. Its FAQ says a submission "doesn't guarantee indexing", and each engine decides for itself (IndexNow: FAQ).

What actually gets a page indexed faster

Announcing a page gets it crawled. Whether it's kept depends on the page. Google's own words:

"Our systems prioritize the fast inclusion of high quality, useful content." (Google Search Central: Ask Google to recrawl your URLs)

In practice: link the new page from a page Google already visits, make sure nothing blocks it (a stray noindex tag is the classic one; see indexing blockers), and give it content that isn't a copy of another page on your site. Typical timelines are in how long Google takes to index a page.

Example

Illustrative cases with round numbers.

A landscaper on Wix publishes a new service page

Wix updates the sitemap and, where IndexNow is on, tells Bing. The owner adds the page to the Services menu and clicks Request indexing once. Nothing else to automate.

A dental office is offered "guaranteed Google indexing" for $50 a month

The tool asks for a Google Cloud key to use the Indexing API. The office's pages are services and blog posts, not job ads or livestreams, so the tool is using the API for pages Google says it doesn't support. The office skips it and checks its sitemap and robots.txt instead.

A plumbing company posts 30 city pages at once

The sitemap lists all 30 and IndexNow sends them to Bing. Two weeks later Search Console shows some as "Discovered, currently not indexed". Automation did its job; the pages are near copies of each other, which is the real problem.

FAQ

Does IndexNow work for Google?

No. Google isn't on IndexNow's list of participating engines (IndexNow: searchengines.json). Use a sitemap and links for Google.

Is the Indexing API against Google's rules for normal pages?

Google says the API "can only be used" for job postings and livestream videos. Other uses are outside that, and abuse may get access revoked.

Do I need an SEO plugin for automatic indexing?

Not usually. Most builders already make a sitemap. A plugin helps if your platform lacks a sitemap or an IndexNow switch.

Why are my pages crawled but not indexed?

Google read them and chose not to keep them, often because of thin or duplicate content. See discovered, currently not indexed.

Different for you?

Keep learning

NextData-driven SEO platforms: which data to trust before you buyWhich SEO numbers are measured and which are estimates, how accurate each kind is, and what a paid SEO data platform adds over free Google tools.

Do it yourself

AI SEO agent review: 8 checks before you chooseEight tests that show whether an AI SEO agent does real work on your site or only sends reports, using your own Search Console and website accounts.

Ask us one question

One line is enough. We reply by email.

Sources

  1. Google Search Central: In-depth guide to how Google Search works
  2. Google Search Central: Ask Google to recrawl your URLs
  3. Google Search Central: Learn about sitemaps
  4. Google Search Central: Build and submit a sitemap
  5. Google Search Central Blog: Sitemaps ping endpoint is going away (June 2023)
  6. Google Search Central: Indexing API quickstart
  7. Google Search Console API: urlInspection.index.inspect
  8. Google Search Central: Meta tags and attributes that Google supports
  9. IndexNow: FAQ
  10. IndexNow: Participating search engines (searchengines.json)