Which business questions to start a conversation analytics pilot with

Short answer

Starting a conversation analytics pilot with "let's see what we find" is the most common and most expensive mistake. A pilot should start with a concrete decision question: one whose answer can change one leader's decision. For example: "which product should next quarter's ad budget go to?" or "are leads lost because of price, or because our reply is late?"

One pilot needs one or two decision questions, 3–5 fields, 4–6 weeks and a success criterion written in advance. This guide shows how to choose the question, examples of good and weak questions, and a week-by-week pilot plan.

Why "what will we find?" fails

A pilot that starts with an open question usually ends like this: there are interesting charts, everyone says "yes, interesting", and no decision changes. The reason is simple: the fields were picked at random, the results have no owner and nobody said in advance "if this number comes out like this, we will do that".

The aim of a pilot is not to show that analytics "works". It is to show that a decision is made better with it. So the question is built backwards from the decision.

Four marks of a good decision question

1. 1It has an ownerA specific leader is waiting for the answer and is ready to act on it.
2. 2The answer is in conversationsThe answer can be in what customers say. "What price does the competitor charge?" is not answered in conversations; "how often do customers mention the competitor?" is.
3. 3The options are knownThe possible decisions are written down in advance: A, B or do nothing.
4. 4It has a deadlineThe decision will be made within 1–3 months, so the pilot result will not arrive too late.

Example pilot questions

• Sales: "For what reason do leads most often not buy, and does it differ by channel?"
• Marketing: "What do customers from the new campaign expect, and does it match our offer?"
• Product: "Which product or feature do customers ask for that we do not have?"
• Support: "Which problem drives the largest share of repeat contacts?"
• Operations: "At what hours and on what topics do conversations go unanswered?"

Weak questions and how to fix them

1. 1"What do customers think of us?"Too broad. Fix: "What are the top three objections raised by customers who did and did not buy in the last two months?"
2. 2"Are customers satisfied?"Conversations do not measure satisfaction; that needs a survey. Fix: "Which service problems are growing fastest?"
3. 3"Are agents doing a good job?"That is agent evaluation, not conversation analytics. Fix: "Which customer questions does a conversation end without answering?"
4. 4"Why did sales fall?"Conversations explain only part of it. Fix: "How did the loss-reason split change in the weeks sales fell?"

From question to fields

Each decision question needs 3–5 fields. For "why do leads not buy?": "product of interest" (one value), "objection" (may have several values), "loss reason" (one value) and "was a next step agreed" (yes / no / unclear). Add "other" to every value list.

Before writing a field, read 30–50 conversations by hand. It shows whether the value list fits reality and removes the biggest mistake before the pilot begins.

A week-by-week pilot plan

1. 1Week 0: preparationWrite down the decision question, owner, options and success criterion. Read 30–50 conversations by hand; build fields and value lists.
2. 2Week 1: calibrationAnalyse the last few weeks of conversations. Check 50 results by hand: is the value AI picked correct? Fix definitions and values where needed.
3. 3Weeks 2–4: collectionNew conversations flow in. Track the share of "other" and analysis coverage weekly.
4. 4Week 5: readingThe owner reviews the results and reads the evidence conversations for the largest values.
5. 5Week 6: decisionOne of the pre-written options is chosen, or "not enough data" is stated — also an honest result.

Write the success criterion first

The success criterion is not the accuracy of the analytics but its effect on the decision. A three-level criterion works well:

• Minimum: the fields work, most results are correct on manual review, the share of "other" is manageable.
• Target: the decision question got a clear answer and the owner decided based on it.
• Excellent: the effect of the decision could be measured during or right after the pilot.

Common pilot problems

• Too few conversations: if the chosen segment has only a few dozen a week, extend the period or widen the segment.
• A field that is too broad: "customer problem" covers everything and says nothing.
• The value list is in your internal language, not the customer's.
• The owner changes mid-pilot, and the new owner has a different question.
• The pilot result becomes "let's analyse everything", and the answer to the first question is forgotten.

An illustrative pilot

This example is illustrative, not a real customer case. A car rental company's sales lead asks: "Are customers who do not book lost because of price, or because of the deposit?" Options: change the price, change the deposit rule, do nothing.

Four fields are set up: car class, objection, loss reason, rental length. After five weeks it is clear that the deposit is mentioned as a loss reason more often than price, especially for short rentals. The owner reads the evidence conversations and decides on a lower-deposit test rate for short rentals. The effect will be checked against the booking share next month.

How a pilot starts in Vexvon

In Vexvon a pilot's fields are set up in the company's panel: name, instruction, closed value list and whether the field takes one value or several. The analysis module runs only after the company switches it on itself; on the admin side the last week, last month or the whole history can be sent for analysis, and after a field changes, old conversations can be re-analysed. The panel shows the number of analysed, queued and failed conversations and the coverage rate. More: Vexvon analytics.

Next step

This week, hold a 30-minute meeting with one leader and write one sentence: "If this number comes out like this, we will ...". For the concept see what is conversation analytics; for the weekly output format see 5 weekly signals for CEOs. We can plan the pilot on your data in a demo.