When AI Answers Wrongly: Managing Confident Mistakes

What AI hallucination is and how to limit it: narrowing scope, citing sources, handover thresholds, forbidden topics and error tracking.

A person hesitates when they do not know; you hear it in their voice. A language model does not hesitate. It says what it does not know in exactly the tone it says what it does — and the customer cannot tell the difference.

Why it happens

Language models do not retrieve answers from a database; they predict the next word. That is why a question with no answer in your knowledge base can still produce a fluent, plausible reply. The problem is not that the model “lies” but that it has no natural mechanism for expressing uncertainty.

Risk varies by topic

Low risk

Opening hours, address, general product features. If wrong, it is corrected; the cost is limited.

Medium risk

Delivery times, stock status, campaign terms. Wrong information creates expectations and turns into complaints.

High risk

Price commitments, returns and warranty scope, contract interpretation, health or financial advice. A wrong answer here can bind the business.

Design against that split: free rein on low-risk topics, and on high-risk ones the assistant should not speak at all — it should hand over.

Six measures that limit the risk

  1. Write the scope — especially what it will not do. Items like “do not commit to prices, do not interpret returns policy, do not give legal assessments” belong explicitly in the system prompt.
  2. Instruct it not to guess. One sentence telling it to hand over when unsure is the single intervention that prevents the most errors.
  3. Cite sources. When the document behind an answer is visible, both agent and customer can verify it. An answer without a source should be treated as suspect.
  4. Keep the knowledge base narrow and current. The most common cause of hallucination is a gap in it. If an old price list is still there, the model will quote it.
  5. Resolve contradictions. When two documents disagree, the model picks one at random.
  6. Keep the handover threshold low. Passing to a human on doubt is always cheaper than a wrong answer.

What to do when you find an error

Wrong answers are inevitable; what matters is what follows. A loop that works:

  1. Let agents flag it with one click. Nobody fills in an error-report form.
  2. Review weekly. The accumulated list is the only real input to improvement.
  3. Decide on each error: add to the knowledge base, correct an existing document, or add a prohibition to the prompt.
  4. Test the fix. A correction is not complete until the same question has been asked again.
  5. Correct it with the customer if it reached them. Silence costs more trust than the error itself.

Measurement

  • Number of flagged wrong answers and its trend. Zero is not good news — it may mean nobody is looking.
  • Share of answers without a source: replies not grounded in the knowledge base.
  • Post-handover corrections: cases where an agent corrected the assistant.
  • Error distribution by topic: if errors concentrate on one subject, take that subject out of scope.

Who is responsible?

Responsibility for a wrong answer sits with the business that published it, not the model provider. That is why narrow scope, written limits and human handover on critical topics are not preferences but part of the design. “The AI said so” is not an answer you can give a customer.

Frequently asked questions

Would a more capable model end the problem?

It reduces it without ending it. Stronger models hallucinate less but keep the confident tone. The real difference lies in knowledge base quality and scope limits rather than the model.

Should a human check every answer?

On high-risk topics yes; on low-risk ones it is unnecessary and expensive. The right design varies the level of oversight with the risk.

Does admitting it is a bot cost trust?

The opposite. Users work it out eventually, and the trust lost by not saying so is far greater than the trust lost by saying so.

Next step: to see how scope limits and handover thresholds are defined, request a demo.