AI summarization tools: how they work and when to trust them
Short answer AI summarization tools shorten long text in one of two ways: they copy the most important sentences (extractive) or write new ones (abstractive). Chatbots like ChatGPT and Gemini write new ones, which reads better but can add details the source never said. Use summaries to triage. Check every number, name, date and "not" against the original before you act.
Want help applying this to your business? Talk it through on a free 30-minute call.
Book a callHow do AI summarization tools work?
They do one of two things. An extractive summary picks the strongest sentences from your text and shows them word for word. An abstractive summary reads the text and writes new, shorter sentences that try to say the same thing.
Microsoft's documentation for its summarization service defines both:
"Extractive summarization: Produces a summary by extracting salient sentences within the source text, together the positioning information of these sentences." (Microsoft Learn: What is summarization?)
"Abstractive summarization: Generates a summary with concise, coherent sentences or words that aren't verbatim extract sentences from the original source." (Microsoft Learn: What is summarization?)
Think of extractive as a highlighter and abstractive as a colleague retelling a meeting. Chatbots you already know, such as ChatGPT, Gemini and Copilot, work the abstractive way by default.
| Extractive | Abstractive | |
|---|---|---|
| What you get | Real sentences copied from the source | New sentences written by the AI |
| Reads like | Choppy highlights | A smooth paragraph or bullet list |
| Typical error | Leaves out the sentence that mattered, or a quote loses its context | States something the source never said, or gets a number, name or "not" wrong |
| How easy to check | Easy: search the source for each sentence | Harder: each claim needs a look back at the source |
| Where you see it | Some summarizer apps, a "key sentences" mode | ChatGPT, Gemini in Gmail, Copilot, most summarizer apps |
The difference matters because the two fail in different ways. An extractive summary can mislead by what it leaves out. An abstractive one can mislead by what it adds.
Can I trust an AI summary?
Trust it to tell you what a document is about and whether you need to read it. Don't trust it alone for anything you'll sign, pay, quote to a customer or put on your website.
The vendors say so themselves. Microsoft's limits page for its own summarizer warns that rewriting the input in new sentences can lose content:
"However, there can be information or accuracy loss." (Microsoft Learn: Characteristics and limitations for summarization)
Google's help page for Gemini in Gmail, which can turn a long email thread into bullet points, states: "Gemini features may suggest inaccurate or inappropriate information." (Gmail Help: Collaborate with Gemini in Gmail)
Research explains why. A 2020 Google Research study of abstractive summarizers found the models "are highly prone to hallucinate content that is unfaithful to the input document", and its human reviewers found invented content in the summaries of every system tested (Maynez et al., 2020). "Hallucinate" here means the AI states something with confidence that isn't in the source. Pretrained models wrote more faithful summaries than older ones, but none was free of invented content.
Long documents add a second risk. A 2023 Stanford-led study found language models used information at the start and end of a long input best, and performance "significantly degrades" when the key fact sits in the middle (Liu et al., 2023). The study tested question answering, not summaries, but the lesson carries over: a clause on page 14 of a 30-page contract is the one most likely to go missing.
What kinds of mistakes do AI summaries make?
Five show up again and again. Knowing them tells you where to look.
| Mistake | What it looks like | Where to check |
|---|---|---|
| Invented detail | "The client approved the quote" when the email only asked for one | Every claim that leads to an action |
| Wrong number or date | $1,500 becomes $15,000; "30 days" becomes "3 days" | Every figure, deadline and percentage |
| Lost "not" or condition | "Refunds are available" when the source said "only within 14 days, unused" | Words like not, unless, only, except, if |
| Wrong person | One customer's complaint pinned on another, or a promise credited to the wrong side | Names and "who said what" in call notes and threads |
| Missing middle | The summary covers the opening and closing but skips a key section | Long files: scan the middle pages yourself |
Extractive tools can't invent wording, so their errors are mostly omissions like the last row. Abstractive tools can make all five.
How do I catch errors in an AI summary?
Make every point traceable, then spot-check the risky parts. Five minutes is usually enough.
- Ask for the source line. Add to your prompt: "After each point, quote the sentence from the text that supports it." This turns an abstractive summary into one you can check like an extractive one.
- Search for each quote. Use Ctrl+F (Cmd+F on a Mac) in the original. If a quote isn't there word for word, treat the point as unproven.
- Check the high-risk words. Every number, date, dollar amount, name, and every not, unless, only and except.
- Ask what's missing. "What does the text say about cancellations, fees or deadlines that you left out?" A second pass often surfaces the skipped middle.
- Read the original for anything binding. Contracts, leases, insurance terms, tax notices, regulator letters and anything health-related. Use the summary as a map of where to read, not a replacement.
Splitting a long file into sections and summarizing each one also helps, because no key fact sits in the middle of a huge input.
Which jobs are safe to hand to a summarizer?
Match the check to what's at stake. The lower the cost of an error, the less you need to verify.
| Job | Risk if it's wrong | How much to check |
|---|---|---|
| Themes in 200 customer reviews | Low: you're spotting patterns | Read 5–10 reviews behind each theme |
| A 40-message email thread | Medium: you may miss who agreed to what | Check names, dates and any promise |
| Sales or service call notes | Medium: wrong notes reach the customer or the crew | Check prices, dates and next steps before sending |
| An industry report or competitor's page | Medium: you may repeat a wrong figure | Find each figure you reuse in the original |
| A contract, lease, policy or official notice | High: legal or money terms | Read the original; use the summary only to find sections |
For review themes in particular, see how reviews shape buying decisions in online reviews and buying decisions.
Which AI summarization tool should I use?
Start with the one you already have. Most owners don't need a separate summarizer app.
- Built into your email: Gemini in Gmail can condense a thread into bullet points. Google says it "requires an eligible Google Workspace or Google AI plan" (Gmail Help).
- A general chatbot: ChatGPT, Gemini or Copilot. Paste text or upload a file and ask for a summary in your own format. The prompt tips above work in all of them.
- A single-purpose summarizer: handy for one-click article summaries, but it's usually the same kind of model behind a simpler form, with fewer controls over format and checks.
Whichever you pick, check what happens to what you paste in. Google says of Workspace:
"Workspace does not use customer data for training models without customer's prior permission or instruction." (Google Workspace: Generative AI Privacy Hub)
OpenAI's Help Center says personal ChatGPT Free, Plus and Pro accounts share chats to improve its models by default, while Business, Enterprise and Edu workspaces are not used for training by default (OpenAI Help Center: Data controls in ChatGPT). On a personal account, turn off "Improve the model for everyone" before pasting customer emails or contracts. Patient information has its own rules: see is ChatGPT HIPAA compliant?
Example
A roofing company and a supplier contract
The owner pastes a 22-page supplier agreement into a chatbot. The summary says "net-30 payment, 2% discount for early payment". Asked to quote its source lines, the chatbot shows the discount sentence, but a search finds the full clause: the discount applies only to orders over $10,000. The owner reads pages 9–15 in full before signing.
A dental office and 300 Google reviews
The office manager exports reviews and asks for the top five complaint themes, with two quotes each. "Long wait times" comes back first. She reads the quoted reviews and ten more; the theme holds, so she raises it at the staff meeting. Total time: 20 minutes instead of an afternoon.
A cleaning company and call notes
An AI tool summarizes a 30-minute quote call as "weekly cleaning, $150 per visit". The owner checks the transcript before emailing the customer: the customer asked for every two weeks. The fix takes one minute; a wrong quote would have cost the job.
FAQ
What is the difference between extractive and abstractive summarization?
Extractive copies the most important sentences from the source unchanged. Abstractive writes new sentences that restate the meaning, which reads better but can introduce errors the source never contained.
Do AI summarizers make things up?
Abstractive ones can. Research on these systems found invented content in the output of every model tested, and Google and Microsoft both warn their summaries can be inaccurate.
Is it safe to paste client documents into an AI summarizer?
It depends on the account. Business plans from Google and OpenAI don't train on your content by default; personal ChatGPT accounts do unless you turn the setting off.
How long a document can AI summarize?
It varies by tool and plan. Whatever the limit, accuracy drops on long inputs, especially for facts in the middle, so summarize long files section by section.
Can I publish an AI summary on my website?
Yes, once you've checked every fact against the source. Anything you publish is your claim, not the tool's. For keeping AI text accurate and in your voice, see AI content creation.
Different for you?
- Writing the summary that sits at the top of your own pages: TL;DR summary.
- Using chatbots for marketing tasks beyond summaries: ChatGPT for marketing.
- Handling patient or health information: is ChatGPT HIPAA compliant?
- Turning what customers say into pages that rank and get cited by AI: SEO and GEO.