Chatbot Testing Checklist: 50 Cases Before Launch

Why internal testing misses everything

The person who configured the bot cannot test it, for the same reason an author cannot proofread their own writing: they ask the questions the system was built to answer, in the vocabulary the material uses. Customers do neither.

Two rules fix most of this. Test cases must come from real customer messages rather than from imagination, and the person running them should not be the person who built it. A colleague from another department finds more in an hour than the project team finds in a week.

Define what a pass means before you start. For most of the cases below, a correct answer is not the only pass — an honest 'I do not know' plus a route to a person is also a pass, and grading it as a failure is how teams end up pressuring a bot into confident invention.

Family one: happy paths

Ten to fifteen cases, drawn from your most frequent real questions. This family is the one teams over-test, so keep it proportionate.

• The five most common questions from last month's real messages, in the customer's own wording rather than a tidied version.
• The same five asked with typos, abbreviations and no punctuation, which is how they actually arrive.
• A question answered by material added most recently, to confirm updates are live.
• A question whose answer differs by location or plan, to confirm the bot asks rather than assumes.
• A multi-part question — two things in one message — which is common and frequently answered only half.

Family two: ambiguity and context

This family finds more defects than any other and is almost never tested, because it requires a conversation rather than a single message.

• A follow-up fragment: ask a question, then send 'and the other one?'. The bot must resolve it against the previous turn.
• A correction: state something, then say 'no, I meant the other model'. Bots routinely continue with the first answer.
• A subject change mid-conversation, then a return to the first subject.
• An ambiguous term that means two things in your business, to confirm the bot asks instead of guessing.
• A question requiring information the customer has already given, to confirm it is not asked twice.
• A very short message — one word — which is extremely common on messaging channels and frequently mishandled.

Family three: languages

• The same question in each supported language, checking the answer is equivalent rather than merely fluent.
• A message mixing two languages in one sentence, which is normal in multilingual markets.
• A language switch mid-conversation, to confirm detection runs per message rather than per conversation.
• A message in a language you do not support, to confirm the bot replies in a supported one rather than attempting the fourth.
• The fallback message in every language — the most visible untranslated string there is.
• A product or brand name in each language, to confirm it is not helpfully translated.

Family four: unsupported and out of scope

Here the pass criterion is an honest refusal plus a route onward. Grade it that way explicitly, or testers will report correct behaviour as a failure.

• A question your material genuinely does not answer.
• A request for legal, medical or regulated advice.
• A request to compare you with a named competitor.
• A request for a discount, phrased three different ways including one with implied leverage.
• A question about a product you do not sell, and about one you discontinued.
• A prediction question: when will it be back in stock, when will there be a sale.
• A request that needs an identity you cannot verify in chat.

Family five: escalation and handover

• A direct request for a person, which must be honoured immediately and without a counter-offer.
• The same request phrased indirectly: 'is anyone actually there?'.
• An angry message, checking that escalation happens without further qualifying questions.
• A complaint mentioning legal action.
• The same conversation at three in the morning, confirming the out-of-hours path exists and names a concrete time.
• A handover where the operator picks up, confirming the transcript travelled and the bot went quiet.
• The return path: after the operator finishes, confirm the bot does not immediately write over them.

Family six: abuse, edge cases and safety

• An attempt to make the bot ignore its instructions, phrased as a new instruction inside the message.
• An attempt to extract internal material — pricing logic, policies, other customers' information.
• An attempt to get the bot to promise something outside policy.
• A very long message, and a message containing only emoji or a link.
• The same question asked ten times quickly, to confirm rate limiting behaves and does not simply fail.
• A request to take an action the bot should not be able to take, such as cancelling an order.
• A safety-adjacent message, confirming it reaches a path that genuinely exists at that hour.

Running it, and what to record

1. 1Write the expected outcome before running the caseNot the expected words — the expected behaviour. 'Answers from material' or 'declines and offers a person'. Deciding afterwards is how a wrong answer gets marked as acceptable.
2. 2Record the retrieved material, not just the replyA pass produced from the wrong source is a future failure. This is only possible if the system records which entries produced the answer.
3. 3Re-run the whole set after every changeMaterial edits and prompt changes both have side effects. A fix that improves five cases and breaks three is common and invisible without a full re-run.
4. 4Keep the set and grow it from productionEvery customer conversation that went wrong becomes a new case. After six months this set is more valuable than anything you wrote before launch.

What Vexvon gives you for testing

Several of the families above depend on being able to see inside an answer rather than only judging the text. Answer transparency records which knowledge entries produced a given reply, which is what turns 'the bot said something wrong' into 'this entry is wrong' — the difference between a test report and an actionable one.

The context family is supported directly: conversation memory covers the last fifteen messages and query rebuilding turns a fragment such as 'and the other one?' into a full standalone query before retrieval runs. Those two mechanisms are exactly what the ambiguity cases are testing, so a failure there points at a specific behaviour rather than at the model generally.

For the escalation family, the paths are explicit and logged: a request for an operator raises a notification, abusive language acts as a stop condition, an agent's stop character pauses the AI for thirty minutes, and the AI can be switched off per conversation. The return-path case is covered by that same thirty-minute pause.

Test conversations are excluded from reporting by default, so a testing week does not distort the month's figures — which matters more than it sounds, because the alternative is teams avoiding thorough testing to keep the dashboard clean.

Frequently asked questions

1. 1How many test cases does a chatbot need before launch?Around fifty, spread across six families: happy paths, ambiguity and context, languages, out-of-scope requests, escalation, and abuse and edge cases. The proportion matters more than the total — most teams over-test the first family and skip the second.
2. 2Who should run the tests?Not the person who configured the bot. They will unconsciously ask in the vocabulary the material uses. A colleague from another department finds more in an hour.
3. 3What counts as a pass?Either a correct answer from the material or an honest refusal plus a route to a person. Grading refusals as failures is how teams pressure a bot into confident invention.
4. 4Which family finds the most defects?Ambiguity and context — follow-up fragments, corrections, subject changes. It is also the family almost nobody tests, because it requires a conversation rather than a single message.
5. 5How often should the set be re-run?After every material or prompt change. Side effects are common: a fix that improves five cases and breaks three is invisible without a full re-run.
6. 6What should be recorded per case?The expected behaviour written in advance, the actual reply, and which material produced it. Without the third, a pass from the wrong source looks identical to a real pass.

Build the set from real messages

Do not invent test cases. Export last month's messages, pick the twenty most frequent and the ten strangest, and use them verbatim including the typos. That set will be more demanding than anything the project team would have written, and it is already an accurate description of what your customers will do on the first day.