The voice A/B test that picked the worse agent, and won by 4 points

We ran a clean A/B test between two versions of a phone agent. Variant B won by 4 points on our success metric. We shipped B. Two weeks later the escalation rate to human agents had gone up, and the "won by 4 points" version was the reason. The test wasn't rigged. It was just the wrong shape for voice, and I'd built it out of chatbot habits.

Here's what I got wrong, in order.

Week 0: the setup that felt correct

For a chatbot A/B test the recipe is boring and reliable. Split traffic, hold everything constant except the one change, define a success metric (task completion, thumbs up, whatever), run until you have significance, ship the winner. I've done it dozens of times and it works, because a text turn is atomic. The user sends a message, the bot sends a message, and nothing happens in between, because there is no "in between."

A voice turn has an in-between. That gap is where this whole story happens, and I designed the test as if the gap didn't exist.

Week 1: what the 4 points actually measured

Here is the detail I glossed over when I set it up. Variant A and Variant B were two different agent builds, and they did not carry the same turn-detection config. B's build had a shorter endpointing threshold: it decided the caller was done talking after about 500ms of silence, where A waited around 800ms. I thought of that as a latency tweak. It is not. It changes who the agent is.

The shorter threshold did two things from one cause. It made B start answering sooner after the caller stopped, which felt snappy. It also made B treat a mid-sentence pause, the breath someone takes in the middle of "I want to cancel my... order from last week," as the end of the turn. So B interrupted people. It answered a question the caller hadn't finished asking.

And our metric couldn't see it. The callers who got cut off but whose intent was already clear still had the task marked complete, so they scored as wins. The callers who got cut off, had to repeat themselves, got annoyed, and asked for a human? A lot of those escalations happened after the task field had already flipped to done, so the metric never counted them. B scored higher on the number while quietly losing more callers, and the gap between those two facts was invisible in the dashboard.

Endpointing is a real variable in a voice test. A text A/B never has to think about it, because text turns have no silences to measure. My A/B test held the prompt constant and let endpointing float between the two builds, so I was changing two things and crediting the result to one.

Week 2: the confounds text doesn't have

Once I started pulling call recordings instead of trusting the scalar, the list of voice-only confounds got long:

• Endpointing, the one that bit us. A 500ms silence threshold and an 800ms threshold are two different agents even with an identical prompt. If it differs between variants, it is part of your experiment whether you meant it to be.
• Barge-in. What happens when the human talks over the agent? Cut off cleanly, keep going, or the two talk over each other for a beat. None of that shows up in a text metric.
• Latency distribution, not the average. A small mean difference can hide a tail: some responses took 1.5s, and on a live call 1.5s of silence feels like the line dropped. People start saying "hello? are you there?" and the transcript fills with noise that then confuses the agent.
• When you score the call. A text conversation ends and then you score it. With voice, the "task done" moment and the "caller gave up" moment can be seconds apart, and the bad part usually comes second.

None of these are prompt content. All of them can differ between two builds without anyone deciding they should.

What I'd measure instead

The fix isn't a better single number. It is treating a voice interaction as a timed, two-party process and measuring it like one.

Start by pinning the turn-taking config across variants the same way you pin the prompt. Endpoint threshold, barge-in policy, VAD settings: fix them, or you are A/B testing them by accident, which is exactly what I did. Then add interruption rate as a first-class metric, because task completion alone told me B was better and interruption rate would have told me the truth: count how often the agent started speaking while the caller was still talking. Report the latency distribution (p50, p95, p99) rather than the mean, since the tail is what makes a call feel broken. Score the call from the recording after the last turn, not at the instant a task field flips true, so the escalation eight seconds later is part of the result. And profile the timing behavior offline before you split live traffic: frameworks like Pipecat and LiveKit let you replay recorded audio through the pipeline, which is the closest thing voice has to a fixed test fixture.

That last one is the chatbot habit I miss most. In text you can freeze the input and get a deterministic comparison for free. In voice you have to manufacture that determinism on purpose, and if you skip it, the timing noise picks your winner for you.

What shipped, and what I'd tell past me

We rolled B back, pinned the endpointing config so both variants waited the same 800ms, and re-ran with interruption rate and tail latency as gates alongside task completion. The winner flipped. A modest version A that waited a beat longer and interrupted less kept more callers to the end.

What I'd tell the version of me who set up that first test: the 4-point win was real, it just measured a different agent than the one I thought I was comparing, because the two builds disagreed about when a caller was finished talking. In text, holding the prompt constant is enough to hold the experiment constant. In voice, the silences between words are part of the agent's behavior, so if you don't pin the timing, it varies on its own and takes your result with it.

Still open for me: I don't have a clean way to put "that interruption felt rude" on a scale. Task completion and interruption count are proxies for it, not the thing itself. If you've found a measurable stand-in for how an interruption actually lands with a caller, I'd genuinely like to hear it.