Hallucination

Because a language model predicts the most likely answer instead of looking up a fact, it can sometimes write down something that sounds plausible but is wrong: a fabricated source reference, a wrong amount, a rule that does not exist. The annoying part is that it sounds just as self-assured as with a correct answer.

Hallucinations cannot be entirely eliminated, but they can be strongly contained. Have the model draw on real sources (RAG), trace answers back to that source, and have a human keep watch in the places where a mistake is costly (see human-in-the-loop).

For us this is exactly why you do not simply let AI loose on customer communication or compliance. The question is not whether a model ever makes a mistake, but whether your process catches that mistake before it goes out.