Publish your failure rate: why we benchmark before we ship

In September, our voice agent failed in a sales demo. Then it failed again in a different demo. By the third one, the message from the field was blunt: “still a trainwreck.”

Here is the uncomfortable part. The agent was not broken. Across every voice call we handled that month, about 3.6% carried a caller-visible defect. That is not great, but it is not a trainwreck either. From the sales seat, though, the hit rate was closer to one in five. Both numbers were true at the same time.

That gap is the subject of this post. It is why we now publish a reliability number for every voice release, before it ships, and why I think every agentic AI vendor should do the same.

Why did 3.6% feel like 20%?

Because failures were not evenly spread. They clustered on one scenario: an employee asks for a personal payroll figure, then asks the agent to do some arithmetic on it. That is the exact scenario sales runs in every demo, because it is the most impressive thing the agent does. The demo script had found the weakest path in the system and was walking down it ten times a day.

When we pulled the logs apart, we found two structural defects underneath. One was a race condition at call teardown: the agent would refuse to recover state because the call had already started closing. The other was a late or fabricated call record, which made some calls invisible for about an hour and reported durations that were plainly wrong. Neither defect cared about the demo script. They just happened to show up there most.

The first fix went in the same night. An engineer raised the endpointing threshold on our telephony provider from 300 ms to 1000 ms. It was validated on one call. Two days later the field reported the same failures. The fix had not held, and we had no way to know, because one call is not evidence.

Every engineering leader has been in this meeting. Sales says the product is unreliable. Engineering says it works in testing. Both are right, and the conversation goes nowhere, because “works” is a binary word for a probabilistic system.

A voice agent is a chain of probabilistic components. Speech-to-text, turn detection, the language model, tool calls into the HR system, text-to-speech, and the telephony layer that holds it all together. Each link has its own failure rate. Multiply them and you get a number. That number is never zero. Pretending it is zero is how you end up with a sales team that feels lied to.

Last week our QA team filed two bugs from a release sanity run that make the point better than I can. One: the same voice query returns different responses across runs. Two: “What is my salary?” is denied on the first ask and answered in full on the second. Identical question, contradictory answers.

You cannot fix “intermittent” with a code review. You can only fix it with a number, a baseline, and a harness that runs the same scenarios enough times to tell signal from noise.

Starting with the next voice release, no voice build ships without a published reliability score. Three parts.

A fixed scenario set. We took the scenarios that actually matter to customers, including the payroll-plus-arithmetic one that embarrassed us, and froze them. Twenty-odd scripted conversations that cover lookups, calculations, denials, handoffs, and hangups. The set only grows. It never gets edited to make a release look better.

Enough runs to mean something. Each scenario runs many times per build, not once. If the failure rate on a scenario is 5%, one run tells you nothing. Thirty runs tell you something. We report per-scenario pass rates and an aggregate, and we track the tail, not just the average. A system that is 99% fine and 1% silent hangup is a worse product than one that is 97% fine and 3% “let me try that again.”

A number sales can quote. The output is a short table that goes to the field with the release. Scenario, pass rate, known failure modes, what changed since the last build. If the payroll-arithmetic path is at 94%, sales knows that. They can choose to demo it or not. What they will not do is find out live in front of a prospect.

Publishing a failure rate feels like handing a competitor a stick to hit you with. I understand the instinct. I think it is wrong for three reasons.

First, the buyer already knows. Every enterprise that has piloted an AI agent has seen it fail. A vendor who says “it just works” is either new or not being straight. A vendor who says “94% on this path, here are the failure modes, here is the trend” sounds like someone who has run this in production. Because they have.

Second, a published 94% beats an implied 100% that turns into 80% in week two. The first number sets expectations. The second destroys trust, and trust is the only thing an enterprise AI vendor actually sells.

Third, the number forces the engineering discipline. If we had owned a reliability harness in September, the 300-to-1000 ms endpointing change would have been tested across a hundred calls, not one, and we would have learned in an hour that it did not hold, instead of in two days from an angry Slack message.

No release without a number. Not “QA passed.” Not “it worked in staging.” A pass rate, per scenario, over enough runs to trust, written down where the people who sell the product can read it.

This is more work. It slows the first release and speeds up every one after it. It also changes the conversation between sales and engineering from “does it work” to “what is the number and which way is it moving.” That is a conversation you can actually win.

Agentic AI is going to split into vendors who demo well and vendors who deploy well. The second group will be the ones who were willing to put a number on the table before anyone asked.

Book a demo