Products SEO & GEO Agent Paid Ads Agent Website Agent Conversion Agent
How it works Pricing FAQ Blog Book a call

Optimize your website for LLMs: crawlers, schema and FAQs

Updated · 8 min read

Short answer To optimize a website for LLMs, let their search bots in and give them plain text. Allow OAI-SearchBot, PerplexityBot, Googlebot and Bingbot through robots.txt and your firewall, get indexed in Google and Bing, and keep key facts in the page's HTML, not only in JavaScript or images. Schema and FAQ markup are optional; Google ignores llms.txt.

Want help applying this to your business? Talk it through on a free 30-minute call.

Book a call

How do LLMs reach your website?

Mostly through a search system that has already fetched your pages. Google's AI answers read Google's index, Copilot reads Bing's, and ChatGPT, Perplexity and Claude send their own search bots. Google's AI optimization guide:

"The way Google Search finds and processes your pages remains the core of how our AI systems access your data." (Google Search Central: Optimizing for generative AI features)
AI answerWhere it reads your siteWhat must be true
Google AI Overviews and AI ModeGoogle's search indexPage indexed, snippets allowed, Search generative AI set to Include
Microsoft Copilot, Bing AI summariesBing's indexBingbot allowed, page indexed in Bing
ChatGPT searchOAI-SearchBotBot allowed by robots.txt and your firewall
PerplexityPerplexityBotSame
ClaudeClaude-SearchBotSame

Microsoft says Bing and Copilot "rely on the same core crawling, indexing, and ranking foundation as traditional search" (Bing Webmaster Guidelines). The bot names come from each company's crawler page.

Step 1: Let AI search bots past robots.txt and your firewall

Open yourdomain.com/robots.txt. Look for a "Disallow: /" line under OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot, Bingbot or "User-agent: *" (all bots). Blocking training bots such as GPTBot and ClaudeBot doesn't take you out of answers. Which bot does what: should you block AI crawlers?

Then check your firewall, because a bot it stops never reads robots.txt. Google's list of basics for AI features includes allowing crawling "by any CDN or hosting infrastructure". OpenAI recommends "allowing requests from our published IP ranges" for OAI-SearchBot, and Perplexity asks the same for PerplexityBot. Two common traps:

  • "Verify you are human" pages. Anthropic says its bots "will not attempt to bypass CAPTCHAs." A challenge page stops them.
  • Cloudflare's Bot Fight Mode. Perplexity's docs warn it "can still challenge Perplexity bots" even after you add a rule allowing them.

Changes take time. OpenAI says about 24 hours after a robots.txt update.

Step 2: Get indexed in Google and Bing

Google's AI answers only link pages that are indexed and "eligible to be shown in Google Search with a snippet" (Google Search Central: AI features and your website). Copilot draws on Bing's index. So:

  1. Verify your site in Google Search Console and in Bing Webmaster Tools, and submit your XML sitemap to both. Bing fetches a new sitemap at once, then "typically at least once per day" (Bing Webmaster Blog, July 2025).
  2. Run URL Inspection on your main pages in each tool. Bing's Live URL tab shows "exactly what the Bingbot sees."
  3. Turn on IndexNow if your website platform offers it. Bing says it tells "Bing and participating search engines when individual URLs are added, updated, or removed."
  4. In Search Console, open Settings, then Search generative AI. Include is the default. Exclude removes your links and content from AI Overviews, AI Mode and AI features in Google Discover. By August 31, 2026, Google had rolled this setting out to every site worldwide, so make sure nobody switched it off.

Pages missing from Google: website not showing up on Google and crawled or discovered, currently not indexed.

Step 3: Put key facts in the HTML, not only in JavaScript or images

A quick test: open a service page, right-click, choose View page source, and search it (Ctrl+F, or Cmd+F on a Mac) for your phone number, a service name and a price. If they aren't in the source, JavaScript draws them in after the page loads, and some AI crawlers may never see them.

  • Google: server-side or pre-rendering "is still a great idea because it makes your website faster for users and crawlers, and not all bots can run JavaScript" (Google Search Central: JavaScript SEO basics).
  • Bing lists "Hiding critical content behind client-side rendering" among things to avoid.
  • A December 2024 test by Vercel, a web host, found that OpenAI's, Anthropic's and Perplexity's crawlers didn't run JavaScript; Googlebot did. Their crawler pages say nothing on JavaScript today, so plan as if they still don't.

The fix is server-side rendering or pre-rendering for main content, a developer's job on a custom site. Chat widgets and review carousels can stay in JavaScript.

Text inside images has the same problem. Google's checklist includes "Making sure that important content is available in textual form," and Bing says images and video "should not be the sole source of information." A price list in a photo, a menu saved as a JPG, a service area shown only on a map: type them on the page.

Step 4: Check for tags that keep you out of AI answers

Meta robots tags, set in code or by an SEO plugin, can limit what AI answers use. In View page source, search for "robots" and "snippet".

TagGoogle AI Overviews and AI ModeCopilot
noindexOut of Google Search, AI features tooOut of Bing search and Copilot
nosnippetNo snippet, so not eligible as a supporting link"may limit Copilot citation quality"
data-nosnippet, max-snippetLimits the text Google may showdata-nosnippet: same as nosnippet
noarchiveNot on Google's list of AI controls"prevents content from being used in Copilot responses"
nocacheNot on Google's list of AI controlsCopilot gets only the URL, title and snippet

Sources: Google Search Central (AI features and your website) and Bing Webmaster Guidelines. On a private page these tags are fine. On a service page they're usually leftovers from a developer or plugin.

Is schema markup needed for LLMs?

No. Google:

"Structured data isn't required for generative AI search, and there's no special schema.org markup you need to add." (Google Search Central: Optimizing for generative AI features)

Microsoft is warmer: structured data "may support clearer grounding but does not guarantee visibility," and it "must accurately reflect visible content" (Bing Webmaster Guidelines). OpenAI's and Perplexity's crawler pages don't mention it. Keep LocalBusiness or Organization markup only if it matches what visitors see. Which types still do something: local business schema markup.

Do FAQ sections help with AI answers?

The visible questions and answers can. FAQ markup no longer earns anything visible on Google, which stopped showing FAQ rich results on May 7, 2026 (Google Search Central: documentation updates). Microsoft, introducing its AI citation report: "Clear headings, tables, and FAQ sections help surface key information and make content easier for AI systems to reference accurately."

What makes an FAQ readable is site setup:

  • In the HTML. Answers must pass the page source test, not load only when someone clicks a question.
  • On the service page. Bing: "URLs focused on a primary topic are more likely to be selected for grounding results." Answer pricing questions on the pricing page, not one catch-all FAQ page.
  • Near the top. Bing: "Place essential information near the top of the URL."

What you can skip: llms.txt and hidden text for AI

  • llms.txt and Markdown copies. Google: "You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them." Keeping one "will neither harm nor help" there. OpenAI's, Perplexity's and Anthropic's crawler pages don't ask site owners for one.
  • Hidden text aimed at AI. Bing warns that content "designed to manipulate or interfere with language models used by Bing or Copilot may result in reduced visibility or removal." Since May 2026 Google's spam policies, which ban hidden text, also cover "attempting to manipulate generative AI responses."

How do you know AI tools can read your site?

  • Copilot: Bing Webmaster Tools' AI Performance report, in public preview since February 2026, shows which pages are cited in Copilot and Bing's AI summaries.
  • Google: Search Console's Generative AI performance report covers its AI features (Google's AI optimization guide).
  • ChatGPT: it adds utm_source=chatgpt.com to the links it sends, so those visits show in your analytics (OpenAI Help Center).

The full staged check: GEO maturity model.

Example

Illustrative cases, not client results.

A landscaper whose prices are a photo

The price page is a scan of the printed flyer. View page source shows no prices, so no crawler can read them. He types his five main prices into a table on the page and keeps the flyer as a download.

A dental clinic behind a bot challenge

A security plugin shows "verify you are human" to visitors it doesn't recognize. PerplexityBot and Claude's bots get the challenge page. The fix: let verified search bots through by their published IP ranges, keep the challenge for everyone else.

A law firm site built as a JavaScript app

The page source holds a title and a script, nothing else. Bing's Live URL view confirms the gap. The developer turns on pre-rendering for the practice-area pages, and the page source test now shows every paragraph.

FAQ

Does page speed matter for AI crawlers?

Google's AI guidance counts page experience, including "reducing latency," among the basics. The OpenAI, Perplexity and Anthropic crawler pages set no speed limit. For visitors, see Core Web Vitals.

How long until changes count?

OpenAI: about 24 hours after a robots.txt change. Google: crawling takes "anywhere from several days to several months." Switching the Search generative AI setting generally takes a few days.

Is this different in Canada?

No. Google's Search generative AI setting applies worldwide, and the OpenAI, Perplexity, Anthropic and Bing crawler rules set no country limits.

Do I need a developer?

For steps 1, 2 and 4, usually not: they're settings in your host, firewall, Search Console and Bing Webmaster Tools. Step 3 often does, if your pages are built with JavaScript.

Different for you?

Keep learning

NextGEO starter plan: four steps for your first monthA first-month GEO plan for a small business: take a baseline, let AI crawlers in, fix your listings, rewrite three pages. Plus what to skip.

Do it yourself

Does ChatGPT recommend your business? A 20-question testA free test any owner can run to see whether ChatGPT names their business, which competitors it names instead, and why.

Ask us one question

One line is enough. We reply by email.

Sources

  1. Google Search Central: Optimizing your website for generative AI features on Google Search
  2. Google Search Central: AI features and your website
  3. Search Console Help: Search generative AI control
  4. Google Search Central: JavaScript SEO basics
  5. Google Search Central: Spam policies for Google web search
  6. Google Search Central: Latest documentation updates (May and June 2026)
  7. OpenAI: Overview of OpenAI crawlers
  8. OpenAI Help Center: Publishers and Developers FAQ
  9. Perplexity Docs: Perplexity crawlers
  10. Claude Help Center: Does Anthropic crawl data from the web, and how can site owners block the crawler?
  11. Bing Webmaster Guidelines
  12. Bing Webmaster Tools Help: URL inspection
  13. Bing Webmaster Blog: Keeping content discoverable with sitemaps in AI-powered search (July 2025)
  14. Bing Webmaster Blog: Introducing AI Performance in Bing Webmaster Tools Public Preview (February 2026)
  15. Vercel: The rise of the AI crawler (December 2024)