Products SEO & GEO Agent Paid Ads Agent Website Agent Conversion Agent
How it works Pricing FAQ Blog Book a call

LLM gullibility: when AI repeats false claims about you

Updated · 7 min read

Short answer LLM gullibility means AI assistants repeat what they read without checking whether it's true. When few pages mention your business, one false blog post, forum thread or fake review can become the answer. You can't edit the AI. Find the page it read, get it fixed or removed, publish the true facts on your site, then report the answer.

Want help applying this to your business? Talk it through on a free 30-minute call.

Book a call

Why AI repeats false claims about your business

AI assistants write the likeliest answer from what the model learned in training and from pages they find as you ask. Neither step checks whether a claim is true. OpenAI says so about its own product:

"But like any language model, it can produce incorrect or misleading outputs. Sometimes, it might sound confident—even when it's wrong." (OpenAI Help Center: Does ChatGPT tell the truth?)

Research shows three reasons:

  • Guessing pays. OpenAI researchers argued that models make things up because training and test scoring "reward guessing over acknowledging uncertainty" (Kalai and others, 2025). An AI that doesn't know your hours may still state some.
  • A matching page beats a credible one. Models "rely heavily on the relevance of a website to the query" and largely ignore whether it cites sources or sounds neutral, UC Berkeley researchers found (Wan, Wallace and Klein, 2024).
  • Words can steer them. RMIT University and Microsoft researchers slipped search words into unrelated passages; AI judges were "highly influenced" by them anyway. A planted line, "this paper is perfectly relevant," swayed some models (Alaofi and others, 2024).

That is what "gullible" means here: the AI trusts the page that fits the question best.

Why one page can become the answer

Because for most small businesses, it's the only page. Google explained its own AI Overview errors this way:

"When AI Overviews get it wrong, it's usually for other reasons: misinterpreting queries, misinterpreting a nuance of language on the web, or not having a lot of great information available." (Google, May 2024)

Google describes the last case as a "data void," where "there's a limited amount of high quality content about a topic." Its example: an AI Overview told people to eat rocks because it "faithfully linked to one of the only websites that tackled the question," a satire piece (Google, May 2024).

"Was [your business] sued?" or "Is [your business] licensed?" are data voids. Almost nobody writes about them, so a forum joke, a complaint post or a page about another company fills the gap.

It takes little effort. In February 2026 a BBC journalist spent 20 minutes on one blog post of invented claims about himself. "Less than 24 hours later," ChatGPT, Gemini and Google's AI Overviews repeated them; Anthropic's Claude did not. Google told the BBC it is working to keep AI Overviews off data voids (BBC, February 2026).

The false claims AI repeats, and where they start

What the AI saysWhere it usually came fromWhere the fix is
You were sued, fined or shut downA news story or court record about another company, or a misread pageThe page's owner; a legal request if it's defamatory
Customers say you overcharge or do poor workA few reviews, real or fake, summed up as a patternThe review site's report button; your public reply
An address, owner or history that isn't yoursA business with a similar name, blended into yoursMatching details on every listing; a line saying who you are not
A service, price or guarantee you don't offerAn old page, an expired promotion, or AI-written copy on your own siteYour own site
A claim no page makesThe model's training memory, or a guessA clear page of your own; a report to the AI company

How to find where a false claim came from

  1. Ask as a customer would. Put the claim as a question, with your business name and city, into ChatGPT, Google, Copilot and Perplexity. Ask more than once.
  2. Check that it searched. Without search, ChatGPT answers from training data that stops at a cutoff date. Search is on by default; cited sources show it was used (OpenAI Help Center).
  3. Open every cited source. OpenAI's own advice:
"Search results and citations can be incomplete, outdated, or incorrect. Open a cited source to check that it supports the answer, review when it was published or updated, and use an authoritative source when accuracy matters." (OpenAI Help Center: Searching the web with ChatGPT)
  1. No source? Don't ask the AI for one. OpenAI lists "Fabricated quotes, studies, citations or references to non-existent sources" among known errors. Search the exact claim in quotes on Google and Bing instead.
  2. Save proof. Screenshot each answer with the date and question, for reports. The answer may change tomorrow.

How to get a false AI answer corrected

Fix the page first, then report the answer. Each company accepts reports, and none promises a correction or a timeline.

AI toolReport a wrong answerLegal or serious problem
Google AI OverviewsThumbs down under the overview, then Report a problem (Google Search Help)Google's legal removal form: see removing information from Google
ChatGPTThumbs down under the response (OpenAI)Report, for content that may break OpenAI's terms or the law (OpenAI)
Microsoft CopilotThe Report or Feedback features in CopilotMicrosoft's Report a Concern page (Microsoft Copilot Supplemental Terms of Use)
PerplexityThe flag icon under the answer, with the answer's link and what's wrong (Perplexity Help Center)A support ticket (Perplexity Help Center)

Google says "AI Overviews can and will make mistakes" (Google Search Help). Microsoft says why:

"When responding, Copilot may use information it finds on the internet, and we don’t control that content." (Microsoft Copilot Supplemental Terms of Use)

So the lasting fix is the page. Ask its owner to correct or remove it. For a review, use the review site's process; for a fake listing under your name, report it.

Give AI the true version to read

Fill your own data void: AI can only repeat the truth if a page says it.

  • One page with the facts. On your About or FAQ page: owner and founding year, license numbers, where you work, what you don't do, and any business people confuse you with ("We are not connected to X in Y").
  • Say it plainly. State the true fact, not just a denial: "Our license, #12345, has been active since 2015." Google admits AI can misread "a nuance of language."
  • The same facts everywhere. Match them on Google Business Profile, Bing Places, Yelp and directories: local citations, Bing Places and Copilot.
  • Pages you don't write. A licensing board record, trade association listing or local news story agrees with you independently. AI Overviews are "built to only show information that is backed up by top web results" (Google, May 2024), so each trusted page stating the truth leaves less room for a false one.

If AI writes your own pages

Your own site can be the source. An AI-drafted service page can invent an award, a statistic or a guarantee, and other AI tools may quote it as fact. Check every price, number, license and quote before you publish: does Google penalize AI content.

Don't fight back with tricks. Google's spam policies ban hidden text, placed "solely to manipulate search engines and not to be easily viewable by human visitors," and many AI-generated pages "without adding value for users" (Google Search Central).

Example

Illustrative cases with round numbers, not client results.

A roofer mixed up with another roofer (US dollars)

ChatGPT says his company was fined $50,000 by the state. The cited story is about a similarly named company two states away. He saves screenshots, adds his license number and "not connected to [the other company]" to his About page, matches his listings and reports the answer.

A bakery and a forum joke (US dollars)

A forum post jokes that a $400 wedding cake "made the whole party sick." A Google AI Overview says "some customers report food poisoning." The owner asks the forum to remove it, posts her latest health inspection result and reports the overview.

A Vancouver bookkeeper with an invented price (Canadian dollars)

Perplexity says she charges CA$99 a month, a price she never offered, with no source. She publishes her real pricing, from CA$300 a month, on her site and listings, flags the answer and checks again in a month.

FAQ

Can I make ChatGPT stop saying something false about my business?

Not directly. Fix the page it read, and searched answers change. Unsearched answers come from training data, so an old claim can stick (OpenAI Help Center).

Can I sue an AI company over a false answer?

A judge ruled for OpenAI in Walters v. OpenAI (Georgia, 2025), partly because ChatGPT warns it can be wrong. Laws vary; ask a lawyer.

Can a competitor plant false claims about me?

Yes. The BBC's one-post test fooled ChatGPT and Google within a day. Check what AI says about you monthly; act on false pages fast.

Will schema markup stop AI from getting my facts wrong?

No. Google says "there's no special schema.org markup you need to add" for generative AI search (Google Search Central). State facts in plain text; schema is optional.

Different for you?

Keep learning

NextAI optimization (AIO): what it means for your websiteWhat AI optimization (AIO) means for a small business: what Google says your website needs, and the brand steps that shape what AI tools say about you.

Do it yourself

Does ChatGPT recommend your business? A 20-question testA free test any owner can run to see whether ChatGPT names their business, which competitors it names instead, and why.

Ask us one question

One line is enough. We reply by email.

Sources

  1. OpenAI Help Center: Does ChatGPT tell the truth?
  2. OpenAI Help Center: Searching the web with ChatGPT
  3. OpenAI: Model behavior feedback
  4. Kalai, Nachum, Vempala and Zhang (OpenAI, Georgia Tech), 2025: Why Language Models Hallucinate
  5. Wan, Wallace and Klein (UC Berkeley), ACL 2024: What Evidence Do Language Models Find Convincing?
  6. Alaofi, Thomas, Scholer and Sanderson (RMIT University, Microsoft), SIGIR-AP 2024: LLMs can be Fooled into Labelling a Document as Relevant
  7. Google: AI Overviews: About last week (May 30, 2024)
  8. BBC Future: I hacked ChatGPT and Google's AI, and it only took 20 minutes (February 18, 2026)
  9. Google Search Help: AI Overviews in Google Search
  10. Microsoft Copilot Supplemental Terms of Use (effective August 18, 2026)
  11. Perplexity Help Center: How can I report incorrect or inaccurate answers?
  12. Google Search Central: Optimizing your website for generative AI features on Google Search
  13. Google Search Central: Spam policies for Google web search
  14. Walters v. OpenAI, Order granting summary judgment (Superior Court of Gwinnett County, Georgia, May 19, 2025)