WebRTC Is Bidirectional — Why Voice Agents Still Talk Over You
Teams shipping voice agents routinely discover that passing WebRTC metrics and passing a "can interrupt" demo still leaves users frustrated by half-heard corrections, phantom history, and false interruptions. The three-layer model and four-state ledger give engineering teams a concrete checklist for what to measure, where state drifts, and which adversarial tests actually predict production quality.
WebRTC's `sendrecv` only guarantees RTP packets flow both ways; it says nothing about who should speak when. A voice agent that interrupts on every "uh-huh" or keeps speaking after a correction has a broken interaction cognition layer, even if its network metrics are perfect. The real failure mode is state divergence: the model cancels a response but the speaker keeps playing it, or playback stops but the conversation history still records text the user never heard. Fixing this requires an event ledger that links each acoustic event to a model action, a playback boundary, and a history correction — not just a faster cancel() call. Four adversarial probes — backchannel retention, entity correction timing, bystander speech filtering, and history truncation after interruption — expose these splits better than any smooth demo. Full-duplex is a scenario capability, not a maturity badge; short-command systems and compliance-critical flows are often safer with explicit turn-taking.
Most "full-duplex" voice agents are actually turn-based pipelines wrapped in a bidirectional transport — audio uploads continuously but gets buffered until the current response finishes, making the interaction still "you finish, then I speak."
The hardest bug to catch is when playback stops correctly but the conversation history retains the unplayed text, causing the next model turn to reason from a shared context that never existed for the user.
Traditional VAD and even semantic end-of-turn detection were designed for turn-taking, not for distinguishing a backchannel "uh-huh" from a takeover "wait, stop" — that classification requires combining speaker identity, task state, keywords, and overlap duration.
An explicit decision object (e.g., `decision = BACKCHANNEL_ACCEPT, confidence = 0.82`) is more valuable than a smarter VAD threshold because it makes misclassifications auditable and recoverable.
Shortening silence detection windows to feel more responsive directly trades off against false end-of-turn triggers during thinking pauses; there is no single knob that optimizes both speed and classification accuracy.
The practical handling of interruption timing dominates the exchange. One view holds that truncating conversation history at the exact millisecond of interruption is unnecessary; a delay of up to 500ms while the current chunk finishes is imperceptible to users and mirrors natural human turn-taking. The alternative question probes whether playback position should come from a player callback or be derived from chunk boundaries, with the firm answer that chunk-based tracking guarantees complete playback.
Got it, chunking is to avoid missing playback. So if the user interrupts right in the middle of a chunk playing, is the history still truncated based on the actual milliseconds played?
Just go to the next window. When the user interrupts, you'll find there's a 500ms delay, and the user won't actually feel it (after all, in real human conversation, it's not like the other person stops talking the instant you open your mouth, right?).
Got it, wait for this chunk to finish playing before switching, sacrificing at most 500ms in exchange for not having to guess the history. Pretty solid.
The heard_until_ms you wrote is critical. In actual integration, is it provided by the player callback, or is it calculated based on audio chunks?
Must be based on chunks, this ensures everything has been played.