From Summarization to Intelligence: Designing Conversation Output You Can Count

We wrote earlier that a call summary should be a schema rather than a paragraph.¹ This article is about what that schema has to survive once a year of conversations is sitting in it and somebody asks a question the schema was not designed for.

Read once, or queried for years

The distinction that organizes everything else is the consumer. A summary has one reader, usually within a day, usually the agent writing wrap-up notes or a supervisor reviewing a complaint. Its job is to be accurate and readable, and prose does that well.

Intelligence has a different consumer: a query, written months later, by someone who was not in the conversation and will never read it. That consumer cannot read. It can group, filter, count, and join. Every design choice that makes output pleasant for the first consumer — nuance, hedging, narrative flow, variable structure — makes it useless to the second.

Both are legitimate and most organizations need both. The failure is producing only the first and planning reports against it.

The prose summary is the least reliable field in the payload

This is worth demonstrating rather than asserting, and the platform documentation obliges. In the published example output for an analyzed call, the generated post-contact summary reads: “The customer and agent’s conversation did not have any clear issues, outcomes or next steps. Agent verified customer information and finished the call.”²

In that same payload, the structured fields tell a different story. An issue is detected on the customer turn “I need to cancel. I want to cancel my plan subscription.” An outcome is detected where the agent applies a discount. An action item is detected where the agent commits to sending details and calling back next week. The category Cancellation matched, with the span that triggered it.²

So the prose says there were no clear issues, outcomes, or next steps, while the typed fields record an issue, an outcome, and an action item. This is a vendor’s own illustrative sample, not a bug report — and that is precisely why it is instructive. The free-text field and the structured fields are produced differently and fail differently, and the one that reads most convincingly is the one you can least rely on.

Free text cannot be counted

Ask for a reason code as free text across a hundred thousand conversations and you will get tens of thousands of distinct strings: “billing issue”, “billing problem”, “issue with bill”, “customer disputed charge”, “charge dispute”. Every one is a reasonable description. Together they are unreportable, and no amount of post-hoc clustering recovers a taxonomy the business will stand behind, because the clusters do not correspond to decisions anybody makes.

The alternative is to declare the vocabulary first. The platform pattern is explicit about this: categories are created before the analysis runs, each carrying between one and twenty rules, with each rule based on one of four criteria — interruptions, keywords, non-talk time, or sentiment — and the output labeled with the category only when the content matches all the rules specified.³ Matches arrive as a list of category names plus the spans that triggered them.²

That shape has three properties worth copying whatever platform you use. The vocabulary is finite, so a count is meaningful. It is declared, so adding a category is a reviewed change rather than a drifting side effect of how a model phrased something. And it is auditable, because the rule that fired is recorded alongside the match.

The most valuable rules detect what was not said

Here is the capability that most clearly separates a declared taxonomy from post-hoc interpretation, and it is routinely overlooked.

Compliance is usually about omission. Did the agent give the required disclosure? Did they state the recording notice in the opening seconds? Did they offer the mandated alternative before closing? These are questions about absence, and a generative summary essentially cannot answer them reliably — a model asked to summarize a call that omitted a disclosure will simply not mention the disclosure, which is indistinguishable from a call where it was given and the summary skipped it.

A declared rule can assert absence directly. The rule criteria include a negation flag that lets you match the presence or absence of keywords, interruptions, silence, or a sentiment within an absolute or relative time range — so “the words welcome or hello were not used in the first ten seconds of the call” is an expressible, countable condition.³

This reframes the compliance build. You are not asking a model to spot violations; you are writing down the script obligations and letting the system report where they were not met. That is a far smaller, far more testable problem, and the output is defensible because the rule is written down.

Every assertion points to a span

An assertion you cannot locate is an assertion you cannot act on. “This call contained a compliance risk” sends a supervisor to listen to eleven minutes of audio. “Compliance risk at 7:42–7:51” sends them to nine seconds.

Good platform output builds this in rather than bolting it on. Detected issues, outcomes, and action items each carry character offsets into the turn that produced them; category matches carry points of interest as millisecond begin and end offsets; sentiment is reported per period with offsets; each transcript turn carries its own identifier, participant, begin and end offsets.² Even redactions carry millisecond timestamps, so the silence in the audio can be located and, if you want, replaced with a tone.²

Copy that discipline into your own schema. Every extracted field gets a reference to the span it came from, and a field with no span is a field with no provenance — which in practice means a field nobody will trust the first time it is challenged. Evidence binding is also what makes the dataset reviewable: a disputed flag becomes a nine-second listen rather than an argument about model quality.

“Not detected” and “not evaluated” are different states

This is the subtlest failure in the domain and the one most likely to produce a confidently wrong compliance report.

Analysis can be skipped. The published output schema includes a section recording skipped analysis, naming the feature, a reason code such as a quota being exceeded or a safety guideline not being met, and the specific categories and rule identifiers that were not evaluated as a result.²

Now consider a report counting conversations where the PotentialFraud category matched. A consumer that treats absence of the label as absence of the condition will silently exclude every conversation where that category was never assessed. The report will be lower than the truth, it will look plausible, and nothing in the number reveals the gap. This is the exact inverse of the problem people worry about with AI — not a false positive, a false sense of safety.

Sentiment is a number with a grain, not a mood

Sentiment is the field most often misused, because its name suggests a single property of a conversation when the useful version is a shape over time.

Platform output reflects this: an overall sentiment score per participant, plus sentiment by period across quarters of the conversation with offsets for each, plus a categorical sentiment on each individual turn.² In the published sample the customer’s overall score is a modest positive while the quarter-level scores move from negative to positive across the call — a recovery, which is a completely different operational event from a call that was mildly pleasant throughout.²

Averaging that into one number per conversation and then averaging across conversations produces a metric that moves for reasons nobody can explain. The questions worth answering are about shape: where did sentiment turn, did it recover before the call ended, which agents recover negative openings. Those require the per-period series to be preserved, not collapsed at write time. Store the arc; derive the summary statistic in the query, where the grain is explicit.

The countable signals that need no language model at all

Half the operationally useful content of a conversation is not in what was said. Platform output includes talk time per participant, average words per minute per participant, interruption instances with durations and who interrupted, non-talk time instances, and per-turn loudness scores.²

These are cheap, deterministic, and almost entirely ignored in favor of the more impressive generative fields. They should not be, because they are the most reliable columns in the dataset — no extraction error, no hallucination surface, no model version dependency. An agent whose words-per-minute climbs steadily across a shift is a staffing signal. Rising customer interruptions on a product line is a messaging signal. Long silences in the middle of calls on one workflow is usually a systems problem, not a coaching problem.

Live assistance and the dataset are two different systems

There is an architectural fork here that is easy to miss because both branches are described with the same words. Conversation analysis can run during the call, driving agent assist and supervisor alerting, or after it, producing the record. These have different latency budgets, different accuracy requirements, and — in the platform model — different category definitions: a category is created as either a post-call or a real-time category, specified at creation rather than chosen per query.³

The requirements genuinely conflict. Live assistance has milliseconds and partial context: it sees the first ninety seconds of a call, not the resolution, and a prompt that arrives after the moment has passed is worthless. It should favor recall and tolerate false positives, because the cost of a wrong nudge an agent ignores is low. The dataset has the whole conversation, no deadline, and a much higher precision bar, because a false compliance flag that nobody catches becomes a coaching conversation about something that did not happen.

The mistake is serving both from one path, which forces a single compromise that is too slow to help an agent and too noisy to report from. Run both, and then be explicit about something most implementations leave undecided: the two will disagree on the same conversation. A live alert fires on a cancellation risk the post-call analysis, seeing the recovery, does not record.

Decide which one is the record of truth — it should be the post-call analysis — and store the real-time firing as an event rather than a fact about the conversation. Then the disagreement rate becomes a metric worth watching in its own right: live alerts that the full-context analysis does not confirm tell you how much noise your agents are absorbing, which is the number that decides whether they keep paying attention to the assistant at all.

Two artifacts, two sets of handling rules

Conversation analysis typically produces twin outputs: an original analyzed file and a redacted twin, differing only in whether sensitive values are masked, with the payload recording which entity types were requested for redaction and the masking mode used.²

Which twin your dataset reads is a governance decision, not a convenience. Analytics and reporting should consume the redacted artifact by default; the original exists for narrow, purpose-bound access. The redaction metadata belongs in the dataset too, because a consumer needs to know that a field is masked rather than empty — those produce the same null and mean entirely different things.

Two cautions carry over from working with regulated conversation data. Redaction is a control, not a status: a transcript with names masked can still be re-identified by joining it to the contact record that produced it, so the protection has to be designed around purpose rather than assumed from the masking.⁵ And a summary of a regulated conversation is itself a record, subject to the same minimum-necessary reasoning as the conversation — which constrains what a summary is allowed to assert, not just who may read it.⁶

Intelligence is worthless until it joins to an outcome

Everything above produces a clean, well-evidenced, countable record of what happened inside a conversation. On its own that is still not intelligence, because the questions that justify the investment are all about what happened outside it.

Did this customer call back within a week? Did the churn risk flagged on Tuesday churn in March? Does the agent with the best sentiment recovery also have the best first-contact resolution? Did the policy change reduce billing contacts, or move them to chat? Every one of these is a join — conversation to customer, conversation to subsequent conversation, conversation to the commercial outcome that followed.

That middle box is where most programs stall, because resolving one customer across voice, chat, email, and messaging is the same identity-resolution problem that shows up in every other domain, and it does not get easier for being about conversations.⁷ It is also the reason a conversation-intelligence project that never leaves the contact-center platform plateaus: the platform knows about contacts, not customers, and certainly not about what the customer did next.

Design the join keys at the start. A conversation record needs a stable conversation identifier, the resolved customer identifier with a confidence on that resolution, the channel, the agent, and the timestamps — and if your identity resolution is probabilistic, that confidence has to travel with every downstream aggregate rather than being quietly discarded.

Comparable over time means versioned

A conversation dataset is most valuable as a time series, which makes it uniquely vulnerable to its own definitions changing.

Two things move underneath it. The taxonomy evolves — you split a category, retire another, tighten a rule threshold — and every change breaks comparability with prior periods unless it is recorded. And the model producing the generative fields changes, on a lifecycle you do not control, which can shift extraction behavior without any change on your side.⁸

At full coverage, quality assurance becomes a distribution

The headline benefit of automated conversation intelligence is coverage: scoring every eligible conversation rather than the small sample a team can hand-review. That is a genuine change, and it quietly invalidates the measurement model most quality programs were built on.

A sampled QA score is a verdict. Two of an agent’s calls were reviewed this month, both passed, the agent passed. A full-coverage score is a distribution: four hundred conversations, a median, a spread, a tail. The tail is the interesting part and the average is nearly meaningless — an agent with a good mean and three catastrophic calls needs a different intervention from one with a mediocre mean and no outliers, and a sampled program could not tell them apart.

Two consequences follow for the architecture. Scores must be stored per conversation, not aggregated at write time, because the distribution is the product. And evaluation of the system itself has to be per field: a compliance flag and a conversation recap have different criteria and very different costs of being wrong, so grading them with one quality number hides the failures that matter. Weight by consequence rather than by volume, and keep the expected answers as specifications rather than reference strings.⁹

What you are actually building

The move from summarization to intelligence is not a model upgrade. The generative step is the commodity part; it was the commodity part two years ago. What is scarce is a schema somebody governs, a taxonomy the business has agreed to, evidence bound to every assertion, an honest account of what was not evaluated, and join keys that connect a conversation to the customer who had it and the outcome that followed.

Those are data-engineering assets. They outlast the model that populates them, they survive a provider change, and they are the reason one organization can tell you why contact volume moved last quarter while another has a year of beautifully written recaps and no idea.