Hallucination.

A language model has no concept of truth. It continues sequences of words that fit the patterns it learnt, and a fluent, confident tone is a property of language, not a statement about certainty. That is what makes hallucinations awkward: the error does not arrive hesitantly, it arrives in the same assured form as the correct answer. Typical patterns are invented citations in a correct-looking format, plausible but wrong figures, software features that do not exist and legal clauses that never were. They occur strikingly often on niche topics with little training material behind them.

The problem cannot be switched off entirely. Part of it is structural: a system that always answers scores better in the usual benchmarks than one that admits it does not know, so guessing is rewarded. Grounding and retrieval cut the error rate substantially, because the model answers from supplied documents rather than from memory. But they also move the problem along. If the retrieved sources are outdated or contradictory, you get a wrong answer carrying a source link, which looks more trustworthy than a wrong answer without one. Source quality becomes the real lever.

Four measures work in practice. First, retrieval with mandatory citation, so every statement traces back to a supplied passage and unsourced answers stand out immediately. Second, structured output validated against a schema, instead of free text that nobody checks. Third, human sign-off wherever a wrong statement gets expensive: prices, deadlines, legal questions and health topics. Fourth, logging of the questions actually asked, because recurring wrong answers reliably show which information is missing from the sources or is worded too vaguely to be used.

For websites there is a side to this that is easily missed: AI assistants hallucinate about companies too. If a detail is missing, or contradicts itself across the website, the Google Business Profile and directory entries, the model fills the gap with whatever is most likely, and quotes an opening time that never applied or a price nobody ever charged. A concrete example is the price that exists only as a graphic on the page. To a model it is invisible, so it gets estimated. Writing facts out as plain sentences and keeping them identical across every source is the most effective prevention there is.