Hallucination

A hallucination is when a model states something false as if it were fact. It is not lying and it is not a bug in the usual sense. The model's job is to produce plausible text, and when it lacks the real answer, plausible text is exactly what you get.

Why it matters when you are buying or building

This is the risk that turns an AI agent from an asset into a liability. A wrong hold-music message is annoying. A bot that quotes a price you do not offer, promises a same-day appointment you cannot honor, or claims to have taken an action it never took creates a real obligation problem with a real customer.

Hallucinations cluster around a few triggers:

• Gaps. The question is in scope but the answer is not in the knowledge base or prompt.
• Capability questions. The user asks whether the agent can do something. If the prompt never says no, the model tends to say yes.
• Specifics under pressure. Numbers, dates, policy details, and names are where invention shows up first.
• Long conversations. Late in a call the original instructions carry less weight relative to everything said since.

What I have seen in production

The clearest example I have: a receptionist agent told a caller, with complete confidence, that it had blocked their number. It could not block numbers. No tool existed. Nothing in the prompt mentioned blocking at all, and that absence was the whole problem. The caller asked, the model had no rule, and it produced the helpful-sounding answer.

The fix was not a better model. It was an explicit negative capability list in the system prompt: a section headed with the things the agent does NOT have the ability to do, enumerated plainly, plus an instruction to say so directly and offer to take a message. Blocking, refunds, account changes, cancellations, anything with a real-world side effect that has no matching tool. That section now goes into every agent I build.

Three other habits that reduce the rate:

1. Ground it. Retrieval over your own content beats the model's memory every time. Give the agent an explicit "if retrieval returns nothing, say you do not know" rule.
2. Give it an exit. An agent with a clean human handoff path invents less, because "let me get someone" is an available answer.
3. Read transcripts. You will not catch these in testing. Read real conversations weekly for the first month. The bad answers are obvious once you see them, and they are almost never the ones you predicted.

You cannot get the rate to zero. You can make the failure mode be "I do not know" instead of a confident invention, and that is the actual goal.