Optimize your website for LLMs: crawlers, schema and FAQs
Short answer To optimize a website for LLMs, let their search bots in and give them plain text. Allow OAI-SearchBot, PerplexityBot, Googlebot and Bingbot through robots.txt and your firewall, get indexed in Google and Bing, and keep key facts in the page's HTML, not only in JavaScript or images. Schema and FAQ markup are optional; Google ignores llms.txt.
Want help applying this to your business? Talk it through on a free 30-minute call.
Book a callHow do LLMs reach your website?
Mostly through a search system that has already fetched your pages. Google's AI answers read Google's index, Copilot reads Bing's, and ChatGPT, Perplexity and Claude send their own search bots. Google's AI optimization guide:
"The way Google Search finds and processes your pages remains the core of how our AI systems access your data." (Google Search Central: Optimizing for generative AI features)
| AI answer | Where it reads your site | What must be true |
|---|---|---|
| Google AI Overviews and AI Mode | Google's search index | Page indexed, snippets allowed, Search generative AI set to Include |
| Microsoft Copilot, Bing AI summaries | Bing's index | Bingbot allowed, page indexed in Bing |
| ChatGPT search | OAI-SearchBot | Bot allowed by robots.txt and your firewall |
| Perplexity | PerplexityBot | Same |
| Claude | Claude-SearchBot | Same |
Microsoft says Bing and Copilot "rely on the same core crawling, indexing, and ranking foundation as traditional search" (Bing Webmaster Guidelines). The bot names come from each company's crawler page.
Step 1: Let AI search bots past robots.txt and your firewall
Open yourdomain.com/robots.txt. Look for a "Disallow: /" line under OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot, Bingbot or "User-agent: *" (all bots). Blocking training bots such as GPTBot and ClaudeBot doesn't take you out of answers. Which bot does what: should you block AI crawlers?
Then check your firewall, because a bot it stops never reads robots.txt. Google's list of basics for AI features includes allowing crawling "by any CDN or hosting infrastructure". OpenAI recommends "allowing requests from our published IP ranges" for OAI-SearchBot, and Perplexity asks the same for PerplexityBot. Two common traps:
- "Verify you are human" pages. Anthropic says its bots "will not attempt to bypass CAPTCHAs." A challenge page stops them.
- Cloudflare's Bot Fight Mode. Perplexity's docs warn it "can still challenge Perplexity bots" even after you add a rule allowing them.
Changes take time. OpenAI says about 24 hours after a robots.txt update.
Step 2: Get indexed in Google and Bing
Google's AI answers only link pages that are indexed and "eligible to be shown in Google Search with a snippet" (Google Search Central: AI features and your website). Copilot draws on Bing's index. So:
- Verify your site in Google Search Console and in Bing Webmaster Tools, and submit your XML sitemap to both. Bing fetches a new sitemap at once, then "typically at least once per day" (Bing Webmaster Blog, July 2025).
- Run URL Inspection on your main pages in each tool. Bing's Live URL tab shows "exactly what the Bingbot sees."
- Turn on IndexNow if your website platform offers it. Bing says it tells "Bing and participating search engines when individual URLs are added, updated, or removed."
- In Search Console, open Settings, then Search generative AI. Include is the default. Exclude removes your links and content from AI Overviews, AI Mode and AI features in Google Discover. By August 31, 2026, Google had rolled this setting out to every site worldwide, so make sure nobody switched it off.
Pages missing from Google: website not showing up on Google and crawled or discovered, currently not indexed.
Step 3: Put key facts in the HTML, not only in JavaScript or images
A quick test: open a service page, right-click, choose View page source, and search it (Ctrl+F, or Cmd+F on a Mac) for your phone number, a service name and a price. If they aren't in the source, JavaScript draws them in after the page loads, and some AI crawlers may never see them.
- Google: server-side or pre-rendering "is still a great idea because it makes your website faster for users and crawlers, and not all bots can run JavaScript" (Google Search Central: JavaScript SEO basics).
- Bing lists "Hiding critical content behind client-side rendering" among things to avoid.
- A December 2024 test by Vercel, a web host, found that OpenAI's, Anthropic's and Perplexity's crawlers didn't run JavaScript; Googlebot did. Their crawler pages say nothing on JavaScript today, so plan as if they still don't.
The fix is server-side rendering or pre-rendering for main content, a developer's job on a custom site. Chat widgets and review carousels can stay in JavaScript.
Text inside images has the same problem. Google's checklist includes "Making sure that important content is available in textual form," and Bing says images and video "should not be the sole source of information." A price list in a photo, a menu saved as a JPG, a service area shown only on a map: type them on the page.
Step 4: Check for tags that keep you out of AI answers
Meta robots tags, set in code or by an SEO plugin, can limit what AI answers use. In View page source, search for "robots" and "snippet".
| Tag | Google AI Overviews and AI Mode | Copilot |
|---|---|---|
| noindex | Out of Google Search, AI features too | Out of Bing search and Copilot |
| nosnippet | No snippet, so not eligible as a supporting link | "may limit Copilot citation quality" |
| data-nosnippet, max-snippet | Limits the text Google may show | data-nosnippet: same as nosnippet |
| noarchive | Not on Google's list of AI controls | "prevents content from being used in Copilot responses" |
| nocache | Not on Google's list of AI controls | Copilot gets only the URL, title and snippet |
Sources: Google Search Central (AI features and your website) and Bing Webmaster Guidelines. On a private page these tags are fine. On a service page they're usually leftovers from a developer or plugin.
Is schema markup needed for LLMs?
No. Google:
"Structured data isn't required for generative AI search, and there's no special schema.org markup you need to add." (Google Search Central: Optimizing for generative AI features)
Microsoft is warmer: structured data "may support clearer grounding but does not guarantee visibility," and it "must accurately reflect visible content" (Bing Webmaster Guidelines). OpenAI's and Perplexity's crawler pages don't mention it. Keep LocalBusiness or Organization markup only if it matches what visitors see. Which types still do something: local business schema markup.
Do FAQ sections help with AI answers?
The visible questions and answers can. FAQ markup no longer earns anything visible on Google, which stopped showing FAQ rich results on May 7, 2026 (Google Search Central: documentation updates). Microsoft, introducing its AI citation report: "Clear headings, tables, and FAQ sections help surface key information and make content easier for AI systems to reference accurately."
What makes an FAQ readable is site setup:
- In the HTML. Answers must pass the page source test, not load only when someone clicks a question.
- On the service page. Bing: "URLs focused on a primary topic are more likely to be selected for grounding results." Answer pricing questions on the pricing page, not one catch-all FAQ page.
- Near the top. Bing: "Place essential information near the top of the URL."
What you can skip: llms.txt and hidden text for AI
- llms.txt and Markdown copies. Google: "You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them." Keeping one "will neither harm nor help" there. OpenAI's, Perplexity's and Anthropic's crawler pages don't ask site owners for one.
- Hidden text aimed at AI. Bing warns that content "designed to manipulate or interfere with language models used by Bing or Copilot may result in reduced visibility or removal." Since May 2026 Google's spam policies, which ban hidden text, also cover "attempting to manipulate generative AI responses."
How do you know AI tools can read your site?
- Copilot: Bing Webmaster Tools' AI Performance report, in public preview since February 2026, shows which pages are cited in Copilot and Bing's AI summaries.
- Google: Search Console's Generative AI performance report covers its AI features (Google's AI optimization guide).
- ChatGPT: it adds utm_source=chatgpt.com to the links it sends, so those visits show in your analytics (OpenAI Help Center).
The full staged check: GEO maturity model.
Example
Illustrative cases, not client results.
A landscaper whose prices are a photo
The price page is a scan of the printed flyer. View page source shows no prices, so no crawler can read them. He types his five main prices into a table on the page and keeps the flyer as a download.
A dental clinic behind a bot challenge
A security plugin shows "verify you are human" to visitors it doesn't recognize. PerplexityBot and Claude's bots get the challenge page. The fix: let verified search bots through by their published IP ranges, keep the challenge for everyone else.
A law firm site built as a JavaScript app
The page source holds a title and a script, nothing else. Bing's Live URL view confirms the gap. The developer turns on pre-rendering for the practice-area pages, and the page source test now shows every paragraph.
FAQ
Does page speed matter for AI crawlers?
Google's AI guidance counts page experience, including "reducing latency," among the basics. The OpenAI, Perplexity and Anthropic crawler pages set no speed limit. For visitors, see Core Web Vitals.
How long until changes count?
OpenAI: about 24 hours after a robots.txt change. Google: crawling takes "anywhere from several days to several months." Switching the Search generative AI setting generally takes a few days.
Is this different in Canada?
No. Google's Search generative AI setting applies worldwide, and the OpenAI, Perplexity, Anthropic and Bing crawler rules set no country limits.
Do I need a developer?
For steps 1, 2 and 4, usually not: they're settings in your host, firewall, Search Console and Bing Webmaster Tools. Step 3 often does, if your pages are built with JavaScript.
Different for you?
- You're deciding which AI bots to block: should you block AI crawlers?
- Your pages aren't on Google at all: website not showing up on Google
- You want the schema details: local business schema markup
- Copilot gets your hours or address wrong: Bing Places and Copilot
- Your site is built so crawlers see an empty page: Website Agent
- You'd rather have the setup checked for you: SEO & GEO Agent