What is AI call analysis? From recording to agent score

The short answer

AI call analysis is the process that turns a recorded conversation between an agent and a customer into text, separates who said what, summarises what the call was about and checks the agent's behaviour against the company's own criteria. It may end in a score, but the score is not where the value is. The value is in the answer, criterion by criterion, to "what was done, what was not, and which line of the transcript shows it".

Put differently: the recording is raw material, the transcript is its readable form, the summary says what happened and the evaluation says whether that met the standard. A team that does not keep these four layers apart either mistakes a transcript for an evaluation or trusts a score it has no evidence for.

One call, five different outputs

An illustrative example makes the difference visible. It is not a real customer call — it is a scenario built to explain the layers. A customer calls an air-conditioning installer and asks for the price of installation in two rooms and the earliest available date. The agent greets them, asks about the room sizes, quotes a price, and ends the call with "we'll call you back" about the date.

1. 1RecordingA four-minute audio file. On its own it tells nobody anything — unless someone listens, the information inside it goes nowhere.
2. 2TranscriptThe conversation as text, line by line: which words the agent said and which the customer said. Now it can be searched, read and quoted.
3. 3Summary"The customer asked about installation in two rooms and a date; a price was given, no date was agreed." It says what happened, not whether it was good.
4. 4Per-criterion evaluationGreeting — met. Clarifying the need — met. Conditions of the price — partial. Agreeing a next step — missed. Each with a reason and a transcript line.
5. 5ScoreOne number calculated from the step statuses. A manager uses it to choose which call to open — but the number alone does not say what to fix.

Five outputs of one call answer five different questions. An operations head needs the summary, a quality specialist needs the evaluation, and the agent needs the specific missed step and the line that shows it.

What a transcript shows and what it does not

A transcript shows the words that were spoken. Whether the agent quoted a price, what the customer objected to, which step was agreed — all of this is visible in text. But text is not sound. Tone, loudness and intonation cannot be reliably read from a transcript. Google's Quality AI guidance says it plainly: transcript-based evaluation does not perform acoustic analysis, and questions such as "did the agent use an upbeat tone" should not be on the form.

The second limit is who is speaking. On a mono recording the system infers the speaker from the voice, and that can be wrong. If a customer's sentence is attributed to the agent, the evaluation will be wrong too. The third is recognition quality: on a noisy or broken recording some words are misheard. That is why a good system marks an uncertain line instead of hiding it.

A summary is not an evaluation

A summary saves time: a manager reads a four-minute conversation in two sentences. But a summary should be neutral — "the agent handled it well" belongs in the evaluation, not the summary. When the two are mixed, managers read a description of every call and draw conclusions that nobody has checked against a criterion.

A practical rule: the summary records what happened, the evaluation records whether it met the standard. One is a fact, the other a judgement.

The evaluation: the standard is yours

The most important part of AI call analysis is not the model but the standard it measures against. A "good call" differs from one company to the next: at a clinic it is giving the preparation instructions, in sales it is agreeing a next step, in support it is understanding the problem without making the customer repeat it. The company has to write that standard — the system does not invent it.

A well-written criterion describes observable behaviour: "the agent offered an installation date or gave an exact callback time". A poorly written one describes a feeling: "the agent was attentive". The first can be checked in a transcript; the second cannot.

The score: the last number, not the first thing to read

A score is useful for choosing which of many calls to look at. Reading it as the agent's worth is a mistake. Two calls scoring 72 can have completely different problems: one skipped the greeting, the other quoted the customer a wrong price. So after the score, a manager always has to go down to the criteria and the evidence.

• Compare scores only between calls measured with the same version of the same standard
• A score produced from a poor recording should not be used against the agent
• A completed sale does not prove a good call, and a lost sale does not prove an agent error
• An AI score should never be the only basis for pay, discipline or dismissal

Which output to look at for which decision

Mixing the outputs does the most damage at the moment a decision is made. This split shows which layer each role should use:

• Understanding quickly what a call was about — the summary; but no conclusion about the agent is drawn from it
• Knowing exactly what the customer was told (during a complaint, for example) — the transcript, and the recording itself where needed
• Deciding what to coach an agent on — the per-criterion evaluation, the missed steps and their evidence
• Choosing which calls to open first — the score; low-scoring calls move up the review queue
• Checking whether the standard itself works — "not applicable" and "partial" statuses that keep recurring on the same criterion

The last item is often forgotten. If a criterion comes back "not applicable" on half of the calls, the problem may be in how the criterion is written rather than in the agents: either it does not belong to that call type, or its condition was never stated clearly. In that sense analysis lets you check the evaluation system itself, not only the agent.

When AI call analysis does not work

The system does not know what is not in the transcript. An agent saying "I've entered the order" does not prove the order was entered — the CRM record does. Whether a problem was resolved cannot be decided from one call either; that needs the contacts that followed. A customer saying "thank you" is not an accurate measure of satisfaction.

There is also an illusion of scale: analysing every call does not mean every call was evaluated correctly. How many calls were fit for analysis at all has to be measured separately — it is worth reading alongside the audit process in our guide to AI call quality audits.

How this looks in Vexvon Audio Analyzer

Vexvon Audio Analyzer works on uploaded call recordings and shows the five layers above separately:

• Recordings are uploaded as files (mp3, wav, m4a and other common formats); uploading the same file twice triggers a warning
• The transcript separates agent and customer — by channel on a stereo recording, by automatic speaker separation on mono; a line the system is not sure about stays "unknown"
• Lines that were poorly recognised are marked in the transcript
• The summary is written in the reviewer's language, and the transcript can be translated into chosen languages
• Each call is checked against a standard the company writes itself: every step gets a status — met, partial, missed or not applicable — with a comment and references to transcript lines
• A 0–100 score is calculated from the step statuses, with recommendations for the call

The system does not point to seconds in the audio: the evidence is specific transcript lines, and the panel scrolls to them.

First step

Ten to fifteen real recordings and a one-page "good call" standard are enough to start. Write the standard as observable steps, analyse the calls and compare the first results with an experienced quality specialist's own evaluation. Wherever the two disagree is where the criterion needs to be made more precise.

More in this section: call QA strategy. To test it on your own calls, get in touch.

Further reading on this topic: automated call quality assurance, call center scorecard template, evidence-based call scoring.