Stopping Chatbot Hallucinations: What Actually Works in Production

The worst hallucination I have shipped was not a wrong price or a made-up address. It was a receptionist telling callers it had blocked their phone number.

The bot had no such ability. There was no tool, no integration, nothing behind it. A caller asked to be put on the do-not-call list, and the agent said, in a completely natural and confident tone, that it had blocked the number and they would not hear from the business again. It did this more than once before I caught it in the transcripts. The business had no record of any block, so the caller got called again, and now you have an angry person who was explicitly told this would not happen.

That is the shape of the dangerous hallucination. Not a factual slip the customer can sanity-check, but a confident claim about an action taken. Nobody catches it in the moment, because the bot sounds exactly as sure as it does when it is right.

Why it happened

The model was doing what models do. It had a role, "receptionist", and enormous prior knowledge about what receptionists do. Blocking a caller is a plausible receptionist action. The prompt described what the bot could do but said nothing about what it could not. In the absence of a contradiction, plausibility wins.

This is the key mental shift. A model does not know the boundary of its own abilities. It knows the boundary you wrote down. Listing your tools does not implicitly exclude everything else, because the model treats the tool list as capabilities available through a specific mechanism, not as an exhaustive definition of self. You have to state the negative space.

The fix: an explicit "abilities you do NOT have" list

The change that solved it was maybe fifteen lines in the system prompt. It now goes in every agent I build, written before the bot has a chance to invent anything.

# Abilities you do NOT have
You cannot do any of the following. If a caller asks, say plainly
that you are not able to do it, then offer to take a message or
transfer them to someone who can.
- You cannot block, ban, or add anyone to a do-not-call list.
- You cannot cancel, refund, or modify an order, invoice, or payment.
- You cannot look up account history, past jobs, or past invoices.
- You cannot send email, documents, contracts, or photos.
- You cannot change, confirm, or guarantee an appointment time
  beyond what the booking tool returns.
- You cannot speak for the owner or commit the business to anything.
- You cannot give legal, medical, or financial advice.

Never say you have done something unless a tool call returned
success for that exact action in this conversation.

Two things make this work.

First, the items are phrased as concrete actions, not categories. "Do not overstate your capabilities" would have done nothing. "You cannot block a number" is checkable against a specific request.

Second, that last line. It is the general rule underneath all the specific ones, and it catches the cases you did not enumerate. The bot may only report an action as completed when a tool actually completed it. I put that sentence at the very end of the prompt, because the end is where models weight most heavily and I want it to be the last thing read before the conversation begins.

How to build the list without guessing

Do not brainstorm it. Derive it.

Walk the adjacent-role space. Ask what a competent human in this role at this business would be able to do, then subtract everything your bot cannot. Whatever is left over is your list. For a receptionist that is blocking, refunds, account lookups, sending things, promising times, speaking for the owner. For a retail shopping assistant it is order status, returns, price matching, holding inventory.

Read the transcripts for near-misses. Every request the bot fielded that it should have refused is a candidate, even if it handled that one well. Handling it well once is luck, not policy.

Ask the client what would make them angry. This is the most productive question in the kickoff call. Business owners answer it immediately and specifically. "I don't want it telling anyone we can do commercial work." "It must never say we're licensed in Indiana." Those go straight into the list.

Everything else is a layer on top

The abilities list is the single highest-value intervention, but it is one layer. Here is the rest of the stack I run on live agents, roughly in order of value per hour of effort.

Ground answers in a knowledge base and say so

Give the bot a real source of truth and tell it explicitly that business facts come only from there. A model with no retrieval will fill gaps from training data, and for a small business that data is either absent or belongs to a different company with a similar name. Getting the chunks right matters as much as having them; see building a chatbot knowledge base for how I structure documents so the right one actually surfaces.

The prompt line that goes with it:

Business facts (hours, services, pricing, policies, locations) come
only from the knowledge base tool. If the knowledge base does not
contain the answer, you do not know it. Do not answer from general
knowledge about this industry.

Give it an exact sentence for not knowing

"Admit when you don't know" is a weak instruction that produces inconsistent hedging. Handing the model a specific sentence produces a specific behavior, and it makes the failure mode greppable. I can search transcripts for that exact phrase and get a clean list of knowledge gaps every week.

I use something like: "I don't have that in front of me, but I can have someone get back to you. What's the best number?" Note that it does not just decline, it routes. A bot that says "I don't know" and stops is a lost lead. A bot that says "I don't know" and captures a callback is a working funnel, which is the whole point of lead qualification.

Make refusal cheap by making handoff easy

Most hallucinations happen when the bot is cornered. It cannot answer, it has no escape, and it invents. If transferring to a human is one tool call away and the prompt encourages it for anything ambiguous, the pressure to invent drops a lot. Get your transfer path solid, including the detail that the transfer tool must reference the live call's control identifier rather than any hardcoded value, or the transfer fails and the caller is stranded. That is covered in call transfer flows.

Constrain the shape of high-risk answers

For anything with numbers or commitments, constrain the format. Prices come back as published ranges, never as a specific quote. Dates come from the booking tool's response, never from the model's arithmetic. Models are notably bad at date math, and "next Tuesday" computed in-context is wrong often enough to matter. Let the tool return the date string and have the bot read it back.

Verify with real records, not clean test cases

This is where I have been burned outside of chatbots too, and the lesson transfers exactly: verify the wiring, not the logic. Green tests on invented inputs once hid a bug that cost a client eleven leads. For agents that means replaying real conversations through the deployed system rather than trying scripted happy paths. Simulator testing also cannot inject some system variables like caller ID, so a whole class of behavior only appears on a real call. Test-call your own cell before any rollout. More on that discipline in testing voice agents.

Read transcripts every week, forever

There is no substitute. I skim every conversation for the first two weeks of a deployment and then sample weekly. I am looking for three patterns: the bot stating a fact I cannot trace to a document, the bot claiming an action, and the customer pushing back or rephrasing. Each one is either a prompt line or a knowledge base document.

What does not work as well as people hope

Turning the temperature to zero. It makes output more deterministic. It does not make it more truthful. A confidently wrong answer at temperature zero is still confidently wrong, just reproducibly so.

Asking the model to rate its own confidence. Self-reported confidence correlates weakly with correctness and gives you a number that feels like a control but is not one. I have not found it worth the tokens.

Very long prompts full of warnings. Past a certain length the rules start contradicting each other and dilute the ones that matter. Fifteen sharp prohibitions beat two hundred lines of caution. If a rule has never corresponded to a real failure, delete it.

Assuming a better model fixes it. Newer models hallucinate less about facts. They will still cheerfully claim to have taken an action, because that is a role-inference failure and not a knowledge failure. The receptionist that blocked numbers was not running an old model.

FAQ

What is the difference between a hallucination and a wrong answer from the knowledge base? A wrong knowledge base answer is a data problem you can fix by editing a document. A hallucination is the model producing something with no source at all. They need different fixes, so before changing anything, check whether the retrieval actually surfaced the right chunk. Most platforms will show you the retrieved context; use it.

Is there any way to guarantee a bot never hallucinates? Not with a generative model in the loop. What you can guarantee is that certain things are never said, by taking them out of the model's hands: hard-code the disclaimer, gate the send in code, restrict the bot to a fixed set of replies for the highest-risk turns. For anything with legal weight, like consent under TCPA rules, enforce it outside the model as well as inside the prompt.

How do I catch hallucinations before customers do? Weekly transcript review plus one automated check: search for the exact phrases the bot uses when it does not know, and separately search for past-tense action claims ("I've blocked", "I've cancelled", "I've sent"). The second search takes five minutes to set up and is the closest thing to an alarm for the dangerous category.

Does the abilities list make the bot sound negative or unhelpful? Not if you pair each refusal with a route. The list says what the bot cannot do; the next sentence tells it to offer a message or a transfer. Callers do not mind being told a limit. They mind being told something happened that did not.

Should I list abilities the bot does not have but might get later? Yes, and update the list when the tool ships. A prohibition on an ability you plan to add next quarter costs nothing today and prevents three months of confident lying. Removing a line is a two-second edit when the real tool arrives.