Training an AI Assistant: Preparing the Knowledge Base

How to train an AI assistant: preparing a knowledge base, writing a system prompt, limiting scope, testing and fixing wrong answers.

“Which model do you use?” tells you very little about an AI assistant’s quality. What decides it is what you give it. An ordinary model on a good knowledge base is more useful than the best model on a bad one.

1. Write the scope (starting with what it will not do)

Before any training, make two lists:

  • Topics it will answer: opening hours, price ranges, delivery times, return conditions, product features.
  • Topics it will not answer: individual price commitments, legal interpretation, health or financial advice, sharing personal data, resolving complaints.

The second list matters more than the first. An assistant whose limits are unwritten invents answers where its knowledge ends.

2. Prepare the knowledge base

  1. Start with what you have. Help centre articles, FAQs, canned replies, product documentation.
  2. Reduce each document to one topic. A long PDF covering three subjects makes finding the right passage harder.
  3. Use a question-and-answer format. The assistant matches text that resembles the user’s question more effectively.
  4. Remove anything out of date. If an old price list is in the knowledge base, the assistant will quote it.
  5. Resolve contradictions. If two documents disagree, the assistant picks one at random.

3. Write the system prompt

The system prompt is the assistant’s rules of behaviour. A good one covers:

  • Identity: “You are the customer service assistant for company X.”
  • Tone: formal or friendly, and how to address people.
  • What to do when it does not know: “If you are not sure, do not guess; offer to transfer to an agent.” This single sentence prevents more errors than any other.
  • Prohibitions: no negotiating prices, no promising discounts, no legal interpretation.
  • Admitting it is a bot: not hiding it when asked.
  • Length: answer briefly and ask a question if detail is needed.

4. Test it — and not with easy questions

Testing an assistant only with well-formed questions is misleading. Your test list should include:

Edge cases

A question with no answer in the knowledge base. Does it invent, or offer to hand over?

Leading questions

“You promised me 50% off last week, didn’t you?” The assistant must not confirm it.

Ambiguous questions

“How much is this?” It should clarify what is being asked rather than guess.

Multi-part questions

When one message contains two questions, does it answer both?

5. Collect and fix wrong answers

Assistants that skip this step get worse over time. A sustainable loop:

  1. Let agents flag a wrong answer with one click.
  2. Review the flagged answers weekly.
  3. For each one decide: add to the knowledge base, correct an existing document, or add a prohibition to the prompt.
  4. Re-test the same question after the fix.

Common mistakes

  • Crawling the website and stopping there. Automatic crawling is a starting point, not an end; enter the critical information by hand.
  • Uploading everything. Internal documents, old presentations and irrelevant content create noise.
  • Writing a long prompt. A three-page rulebook gets diluted; short, clear rules are followed better.
  • Forgetting the handover. When the assistant hits a dead end, reaching a person must be easy.

Frequently asked questions

How large should the knowledge base be?

Small and accurate beats large and scattered. Writing the 20–30 most frequent topics well is a sufficient start for most businesses.

How often should I update it?

Immediately on any product or price change; otherwise a short monthly review is enough.

Who is responsible if the assistant answers wrongly?

The business is. That is why narrow scope, written limits and handing over on critical topics are part of the design.

Next step: to see the assistant training and knowledge base screens, request a demo.