Maaz Zindani

Loading the index…

↑↓ move↵ openesc close4072 pages

Help for owners

The AI in my app keeps giving wrong or invented answers

Short answer

AI models can produce confident answers that are wrong. Today, limit what the feature may say, add a way to reach a person, and save examples of bad answers. Fixing it means giving the feature the right sources, allowing it to say it does not know, and testing it on real questions.

  • Written for business owners
  • Updated
  • By Maaz Zindani, Senior Software Engineer

Does this sound like your situation?

  • A customer received a wrong answer from the bot.
  • The bot invents prices, policies or links.
  • It answers fine in my tests and badly for users.
  • Answers change from one day to the next.
  • I cannot tell how often it is wrong.

Why does this happen?

  1. Models can state false things confidently

    Anthropic's guidance says even advanced models can generate text that is factually incorrect or inconsistent with the given context.

  2. The bot has no source to rely on

    If it is not given your documents, it answers from general knowledge. The documentation advises restricting it to the information you provide.

  3. It was never told it may say I do not know

    The same guide says explicitly allowing uncertainty can drastically reduce false information.

  4. Nobody measures it

    Without a set of test questions with correct answers, a change can make things worse unseen. Comparing several runs of the same prompt can also reveal inconsistency, according to the same guide.

What can I do about it today?

  1. Save the bad answersScreenshots with the question, the answer and the time. These become your test set.
  2. Narrow what it may talk aboutAsk your developer to limit it to your own topics and to refuse the rest.
  3. Add a visible note and a hand-offSay that answers may be wrong and show how to reach a person.
  4. Switch it off for risky topicsPrices, refunds, medical and legal questions are safer with a human.
  5. Tell affected customers if neededIf a wrong answer cost someone money, speak to your adviser about what you owe them.

When should I bring someone in?

  • The bot has given wrong information about money, health, safety or legal matters.
  • You cannot say how often it is wrong.
  • Fixing prompts has not helped.

How do I handle it?

  1. Collect failures into a test setReal bad answers become test questions with the correct answers written beside them.
  2. Find the causeI check whether the bot lacked the source, ignored it, or was never told to admit gaps.
  3. Apply the documented fixesGround it in your content, allow it to say it does not know, and ask for supporting quotes where suitable.
  4. Re-run the whole setEvery change is checked against all test questions, not only the one that failed.
  5. Monitor after releaseSampled conversations are reviewed and the test set grows with new failures.

The price depends on what I find, so I quote in writing after a short discovery call. Tell me what is happening and I reply within 24 hours.

What do owners ask about this?

Can the wrong answers be removed completely?

No. Anthropic's guide says its techniques significantly reduce them but do not eliminate them, so important answers should still be checked.

Is a better model the answer?

Sometimes, but sources, limits and testing usually matter more. I measure before suggesting a switch.

Who is responsible for what the bot says?

That is a legal question about your business. Speak to a lawyer.

Sources

How was this page made?

I drafted this page with AI assistance from the official documents listed under Sources. Found a mistake? Tell me.

For your developer: the technical fixes

Next step

Want this off your desk?

Send me a few lines on where things stand. I reply within 24 hours.