Chatbot Knowledge Base Audit: Fix This First

Why the material decides the verdict

Buyers evaluate models and then deploy documents. The gap between those two activities is where most disappointment lives. A capable model answering from contradictory, incomplete or stale material produces confident wrong answers, which is a worse outcome than no answer at all.

This is good news, because material is fixable and cheap to fix. The audit below takes a few days for most businesses, requires no engineering, and removes the majority of the answers that would otherwise have to be discovered by customers.

Run it before launch, not after. The cost of a wrong answer found in an audit is an edit; the cost of the same answer found by a customer is the answer plus the trust.

Failure one: gaps

A gap is a question customers ask that the material does not answer. They are invisible from inside the company, because the people writing the material already know the answers and do not think of them as questions.

1. 1Start from real questions, not from your documentsExport six months of support tickets, chat logs, Instagram and WhatsApp messages, and the questions sales gets on first calls. Sort by frequency. This list, not your help centre structure, is the specification.
2. 2Mark each question answerable or notFor each of the top hundred, find the material that answers it. Where you cannot, you have found a gap. Expect to find more than you think — twenty to forty per cent is normal on a first audit.
3. 3Look specifically for the awkward onesPricing edge cases, what you do not support, delivery delays, refunds, comparisons with competitors. These are disproportionately absent because nobody enjoys writing them, and disproportionately asked because customers care about them.
4. 4Write the missing answers in the voice you want the bot to useThe bot will reproduce the register of the material. Material written as internal notes produces answers that read as internal notes.

Failure two: contradictions

A contradiction is two pieces of material that both answer a question, differently. It is more damaging than a gap, because the bot will answer confidently with whichever it retrieved, and the answer will vary between customers.

• The same fact in several documents — delivery time in the FAQ, on the product page and in a PDF, all written at different times.
• Old campaign material that states a price or a promise no longer offered.
• Regional differences written as universal statements, so a rule true in one city is stated as company policy.
• Internal drafts uploaded alongside published copy, which the bot cannot distinguish.
• Translations that drifted, where the Azerbaijani and English versions of a policy no longer say the same thing.

The practical fix is to nominate a single source per fact. For every important claim — price, delivery, coverage, guarantee — decide which document owns it, and make the others reference rather than restate it. This is more valuable than any retrieval tuning.

Failure three: staleness

Material decays quietly. Nothing breaks when a price changes; the old page simply keeps existing and keeps being retrieved.

• Date-stamp everything at import, so age is visible without reading.
• Flag anything not reviewed in twelve months for a decision: confirm, update or remove.
• Remove rather than archive, where the material is public-facing. An archived document that is still indexed is still an answer.
• Identify the fast-decaying categories — pricing, stock, opening hours, staff names, campaign terms — and set a shorter review cycle just for those.
• Name an owner per category. Material without an owner is material nobody updates.

Failure four: no escalation content

The most commonly missing category is not product information. It is the material that tells the bot what to do when it should not answer — and it is missing because it does not feel like knowledge.

• What the bot should say when it does not know, in each language. Written, not improvised.
• What is out of scope entirely: legal advice, medical claims, competitor comparisons, anything regulated.
• How to hand over: what to say, what to promise about timing, and what to say when it is out of hours.
• What the bot may never promise — discounts, exceptions, delivery dates outside the published policy.
• The complaint path, written as a response rather than as a process document.

Scoring the audit

1. 1CoverageOf your top hundred real questions, what share has an answer in the material? Below eighty per cent, launch on a narrower scope rather than a broader one.
2. 2ConsistencyHow many of your important facts have more than one source? Every one of those is a future inconsistent answer.
3. 3FreshnessWhat share of material has been reviewed in the last twelve months? Fast-decaying categories should be near total.
4. 4Escalation readinessAre the five escalation documents above written? This is binary and it is usually zero before someone asks.

Score these four before launch and again after ninety days. The second run is more useful than the first, because by then the conversations have shown you which questions you were wrong about.

How Vexvon structures the knowledge base

Material comes from five sources: your site crawled to as many as 4,500 pages, PDFs up to 25 MB, manual or bulk entries at up to 200 per request, a product catalogue with variants and images, and an Instagram archive of previous questions and answers. That last source is directly useful for the gap audit above, because it is a record of what customers actually asked.

Each entry is classified into one of thirteen types — FAQ, pricing, product, catalogue, branch, campaign, opening hours, policy and others — with automatic type detection, and can be activated or deactivated individually. Deactivation rather than deletion is what makes a contradiction fixable without losing the document.

Each entry is also targeted: chatbot, call agent, or both. The same base therefore serves two different AI surfaces without duplicating the material, which removes one of the most common sources of contradiction — a second copy of the same fact maintained separately for the phone.

Retrieval runs over Milvus with six results by default and optional hybrid reranking, each company's material isolated by partition key, and embeddings are refreshed automatically when an entry changes. Answer transparency records which entries produced a given reply, which turns an audit finding from a guess into a specific document to edit.

Frequently asked questions

1. 1What is a chatbot knowledge base audit?A pre-launch review of the material the bot answers from, checking four things: whether real customer questions are covered, whether facts contradict each other, whether the material is current, and whether the bot has been told what to do when it should not answer.
2. 2How long does it take?For most businesses, a few days. The longest part is writing the missing answers, not finding them, and the awkward topics take longest because nobody has written them down before.
3. 3How much coverage is enough to launch?Aim for at least eighty per cent of your top hundred real questions. Below that, narrow the bot's scope rather than launching broadly and hoping.
4. 4What is the most commonly missed item?Escalation content — what the bot says when it does not know, what is out of scope and how handover is worded. It is missing because it does not feel like knowledge.
5. 5How do you handle contradictions?Nominate a single source per fact. Every important claim should be owned by one document, with the others referencing it rather than restating it.
6. 6How often should the audit be repeated?Score it before launch and again after ninety days, then annually — with a shorter cycle for fast-decaying categories such as pricing, stock and opening hours.

Start with a hundred real questions

The whole audit begins with one export: six months of real customer questions from every channel, sorted by frequency. Everything else follows from that list. Teams that start from their existing documentation instead consistently audit the wrong thing, because documentation is organised around what the company wanted to say.