When it is wrong, and how you catch it

What you'll be able to do afterwards: Recognise a likely fabrication before it reaches a customer, and apply a three-question check.

A model will produce a confident sentence citing a report that does not exist. Ask for a statistic and you may get a precise, plausible-looking figure with no basis in anything.

This is not deception. The model has no mechanism for checking whether what it wrote is true — remember lesson 1. It is producing the most likely continuation of your text. If your question invites a number, a plausible number is the most likely continuation. Filling the shape with something convincing is exactly what it was asked to do.

Where it bites most

Fabrication is not random. It clusters, and knowing the clusters is most of the defence.

• Specific numbers. Statistics, market sizes, percentages, dates. Anything with a decimal point arrives with an unearned air of precision.
• Citations and links. Reports, studies, case law, articles. Titles and authors can be invented convincingly, and sometimes point at real people who never wrote it.
• Quotes. Direct quotation of a real person is a frequent fabrication, and a costly one if it is published.
• Obscure specifics. The less common the fact, the weaker the pattern, and the more the model improvises to fill the gap.
• Your own business. Your margins, your customers, your history. It has no source for these at all.

By contrast, it is usually reliable on common knowledge, standard definitions, well-trodden code and ordinary business prose — because the patterns are strong and the specific detail is not required.

The three questions

Apply these to anything that matters. They take seconds and catch most of it.

1. Does this contain a number, a date, or a name I did not supply? If yes, treat it as unverified until you have a source. This single question catches the majority of consequential errors.

2. Could I find this myself in two minutes? If not, that is a signal. A claim you cannot check is a claim you should not publish, no matter how reasonable it sounds.

3. If this is wrong, who finds out and how bad is it? This is the real one. A wrong line in an internal draft is a shrug. A wrong figure in a client proposal is a phone call. A wrong claim in something published is a correction notice. Matching your checking effort to the consequence is the whole skill.

Rules worth adopting as habits

Give it the source rather than asking it to remember. If you need a summary of a report, supply the report. Grounding the task in a document you provided moves it from recall to reading, and reading is far more reliable. This is the single most effective change most people can make.

Never accept a statistic you did not bring. If you supply the numbers, the model can arrange and explain them well. If it supplies them, you have no idea where they came from.

Keep a human on anything published. Not because the model is bad, but because a published error costs more than a review does.

Correct it in the conversation, don't restart. "That figure is wrong, use these numbers instead, and flag any other statistic you were about to invent" works well and often surfaces the next problem.

When it matters, ask it to argue the other side. "What would have to be true for this to be wrong?" produces a genuinely useful critique, because it is now doing the thing it is good at — generating plausible text — in the service of finding weaknesses.

The posture that works

Do not aim for a system that is never wrong. Aim for one where being wrong is cheap and gets caught.

That means: drafts rather than finals, sources you supplied, numbers you brought, a single review step before anything leaves the building, and a correction policy ready for when something slips. Teams that accept the failure mode and design around it get real value. Teams that assume accuracy get one embarrassing moment and abandon the whole thing.

Next: lesson 6, where you stop using AI and start building a workflow.