RAG Chatbot Accuracy: Retrieval Decides the Answer

What retrieval-augmented generation actually does

A retrieval-based chatbot answers in two stages, and only the second one is the model. First the system searches your material for the pieces most likely to contain the answer. Then it hands those pieces to the model and asks it to write a reply using them.

This design exists because the alternative does not work for business. A model answering from its own training has no knowledge of your prices, your policies or your stock, and no way to be corrected when it is wrong. Retrieval makes the answer traceable to a document you control, which is what makes correction a one-line edit rather than a retraining exercise.

The consequence is the point of this article: if the retrieval step hands over the wrong three paragraphs, no model can produce the right answer. Most accuracy work is therefore retrieval work, and most teams spend it on prompts instead.

The four retrieval failures

1. 1Nothing relevant was foundThe answer exists in your material but the search did not surface it, usually because the customer's words and your document's words do not overlap. Customers say 'how fast do you deliver'; the document is titled 'logistics terms'. The correct behaviour here is to say so and offer a person — but many systems answer from whatever they did find.
2. 2Something relevant but wrong was foundThe most dangerous failure, because the answer looks well-sourced. An old price page, a regional exception, a superseded policy. This is a knowledge-base problem surfacing as a model problem.
3. 3The right document was found but the wrong piece of itMaterial is broken into chunks before it is searched. If the chunk boundary falls in the middle of a policy, the retrieved fragment can state a condition without its exception, which produces a confidently incomplete answer.
4. 4Too much was foundHand a model ten loosely related passages and it will blend them. The result is a fluent answer that no single source supports, and that is very hard to trace afterwards.

Chunking, in business terms

Chunking is how your documents are cut up before they are indexed. It sounds like an engineering detail and it changes answers more than almost anything else.

• Chunks that are too small lose the context that makes a statement true — a price without the plan it belongs to, a condition without its exception.
• Chunks that are too large dilute the match, so a highly relevant sentence buried in a long page ranks below a moderately relevant short one.
• Chunks that cut across a natural boundary are the worst case: half a rule retrieved as though it were the whole rule.
• Material written as short, self-contained answers chunks well without any tuning at all. This is the cheapest accuracy improvement available, and it is a writing task rather than a technical one.

The practical implication for a business team: rewriting your ten most important policies as short, self-contained statements will usually improve answers more than any retrieval parameter you could change.

Ranking and why the top result matters

Retrieval returns a ranked list and only the top few are handed to the model. Everything below the cut-off may as well not exist, which makes the ranking the real gatekeeper of accuracy.

• A fixed number of results is a trade-off, not a truth. Too few and a slightly-worded question misses; too many and the model blends unrelated passages.
• Pure semantic search struggles with exact tokens — product codes, model numbers, order references. Hybrid approaches that combine keyword and semantic matching handle those far better.
• Reranking, where a second pass re-orders the initial results, helps most when your material contains many similar documents — which is exactly the situation most companies are in.
• Recency is not automatically part of ranking. A three-year-old page can outrank this month's if it is worded more similarly to the question. That has to be handled in the material, not hoped away.

Grounding: making the answer stay in the material

Grounding is the discipline of answering only from what was retrieved. It is partly a prompt, mostly a policy, and it is what separates a business chatbot from a confident guesser.

• The system must have a defined behaviour when retrieval returns nothing useful: say so, offer a person, and do not improvise.
• The bot should not fill gaps from general world knowledge on business-specific questions. A plausible delivery time it invented is worse than an admission.
• Scope limits belong in the material as well as the prompt: legal advice, medical claims and competitor comparisons should be explicitly out of bounds.
• Answers should be traceable. If the system records which pieces produced a reply, a wrong answer becomes a document to edit. If it does not, it becomes an argument.

How to evaluate retrieval quality

1. 1Build a fixed question setFifty to a hundred real customer questions with known correct answers, including the awkward ones. This set is the instrument; without it every change is an opinion.
2. 2Measure whether the right material was retrieved, separately from whether the answer was rightThese are two different failures with two different fixes. Conflating them is why teams tune prompts to fix knowledge-base problems.
3. 3Include questions that have no answerAt least ten per cent of the set should be things your material genuinely does not cover. You are testing whether the system admits it, which is the behaviour customers judge you on.
4. 4Re-run the whole set after any changeMaterial edits, chunking changes, model changes. A fix that improves ten answers and breaks six is a common and invisible outcome without a fixed set.
5. 5Read the failures, do not just count themThe counts tell you there is a problem; the retrieved passages tell you which of the four failure types it is.

How retrieval works in Vexvon

Retrieval runs over Milvus, returning six results by default, with optional hybrid reranking for material where many documents look alike. Each company's material is isolated by partition key, so retrieval can only reach that company's own documents.

Two mechanisms address the failure types above directly. Query rebuilding turns an incomplete follow-up such as 'what about the red one?' into a full standalone query before retrieval runs — without it, the second question in a conversation retrieves almost nothing useful. And conversation memory of the last fifteen messages keeps the thread coherent across turns.

Grounding is explicit rather than implied. There is a defined fallback when no context is found, the AI stops rather than improvising, and a request for an operator raises a notification. Answer transparency records which knowledge entries produced a given reply, which is what turns the evaluation described above from guesswork into a specific document to edit.

On the material side, entries are typed into thirteen categories, can be activated or deactivated individually, and are targeted at the chatbot, the call agent or both. Embeddings refresh automatically when an entry changes, so a corrected document is a corrected answer without a rebuild.

Frequently asked questions

1. 1What is a RAG chatbot?One that searches your material for relevant passages and then asks a model to answer using only those passages. The point is that answers are traceable to documents you control, so a wrong answer is fixed by an edit rather than by retraining.
2. 2Why does it still give wrong answers?Usually because retrieval handed over the wrong material, not because the model wrote badly. Four failures account for most of it: nothing relevant found, something relevant but outdated, the right document but the wrong fragment, and too much loosely related material blended together.
3. 3Does a better model fix accuracy?Rarely, on business-specific questions. The model can only use what it was given. Improving the material and what gets retrieved has far more effect than changing the model.
4. 4What is chunking and why does it matter?It is how documents are cut up before indexing. Fragments that are too small lose the context that makes a statement true, and fragments that cut across a rule can state a condition without its exception.
5. 5How do we evaluate it?Build a fixed set of fifty to a hundred real questions with known answers, including at least ten per cent that your material does not cover, and re-run the whole set after every change.
6. 6When should the bot refuse to answer?Whenever retrieval returns nothing useful, and on anything out of scope such as legal or medical claims. The correct behaviour is to say so and offer a person.

Start with a question set, not a prompt

If accuracy is the problem you are trying to solve, do not start by editing prompts. Build the fixed question set first. It takes a day, it makes every subsequent change measurable, and in most cases the first run of it identifies the handful of documents responsible for the majority of wrong answers.