WebRTC Is Bidirectional, So Why Do Voice Agents Still Talk Over You?

A voice agent's WebRTC metrics can all be normal, yet it still behaves like a customer service rep who doesn't listen.

The microphone keeps uploading, the speaker keeps downloading—the connection is indeed bidirectional. But the user just says "uh-huh," and the system immediately cuts off the response; the user says "not Hangzhou, Qingdao," but the model has already stopped, and the player still broadcasts half a sentence more; the user hasn't interrupted, but someone nearby speaks, and the backend creates a new turn; a more subtle case is when the sound stops, but the conversation history still records that unheard response, and the next round the model will reason based on a shared context that never existed.

These problems show that "both sides can transmit audio simultaneously" and "both sides can communicate naturally and simultaneously" are not the same capability.

One-Sentence Conclusion

A full-duplex voice agent has at least three layers: the transport layer handles simultaneous media sending and receiving, the inference layer handles letting new input change the output being generated, and the interaction cognition layer handles judging pauses, backchannels, background noise, and valid interruptions. The real production threshold is whether audio capture, model response, playback queue, and conversation history can converge within a single event protocol, not whether WebRTC is enabled in the interface.

1. First, Break Down "Full-Duplex" into Three Different Problems

In engineering discussions, "full-duplex" often refers to three things at once.

The first is transport: can the client send microphone audio while receiving remote audio. The second is inference: when the model has already started generating, can new input enter the current state and change subsequent output. The third is interaction cognition: can the system determine whether an overlapping sound is a backchannel, an interruption, a correction, a call to someone else, or just noise.

Each layer can succeed or fail independently.

When the transport layer succeeds but the inference layer fails, microphone data continuously reaches the server, but the model still waits for the current TTS to finish playing before processing. When the inference layer succeeds but the interaction cognition layer fails, the system has a cancel interface but treats every "uh-huh" and cough as a user takeover. When the interaction judgment is correct but the playback state fails, the model decides to yield the floor, but the client buffer continues to produce sound. When playback indeed stops but the history state fails, the system treats text the user never heard as if it had been spoken.

Therefore, "full-duplex" cannot be a boolean field. A more practical approach is to write it as a three-layer capability declaration and record different evidence and failure modes for each layer.

2. Transport Layer: sendrecv Only Means RTP Can Flow Bidirectionally

The W3C WebRTC specification defines RTCRtpTransceiverDirection as sendrecv, sendonly, recvonly, inactive, and stopped. The meaning of sendrecv is very specific: the sender can send RTP, and the receiver can receive RTP, provided the remote end accepts the negotiation.

This definition answers none of the following: whether the user has finished speaking, whether the system should continue speaking, whether an overlap is an interruption, and who the sound just received belongs to.

What WebRTC excels at is solving media connection problems: tracks, codecs, negotiation, network paths, bandwidth estimation, jitter, and clocks. It allows data in both directions to exist simultaneously and provides observations like packet loss, round-trip time, jitter, and audio energy. Transport layer acceptance testing should focus on connection establishment, uplink continuity, downlink continuity, network degradation, and device switching.

But even if all these metrics pass, the system can still be a turn-based pipeline dressed in a bidirectional channel. The most typical pseudo-full-duplex is: user audio is continuously uploaded, and the server buffers it until the current response is complete before submitting it to the model. The network layer is sendrecv, but the interaction control is still "you finish, then I speak."

Therefore, the first boundary can be stated very clearly: WebRTC is one of the necessary transport conditions for full-duplex voice, not a sufficient condition for natural floor decision-making.

3. Inference Layer: New Input Must Be Able to Change the Output Happening Now

Suppose the agent is saying: "It will rain tomorrow in Hangzhou..." and the user inserts: "Not Hangzhou, Qingdao."

Inference-layer full-duplex is not "the server also received this audio," but whether this correction can change the response currently running. The system must complete at least five actions.

1. Create a trackable event identity for the new input and record its corresponding session moment.
2. Find the response_id currently being generated or played.
3. Decide to continue, pause, cancel, or re-plan, rather than uniformly executing a stop.
4. Make the cancellation act on both the model output and the client playback queue.
5. Correct the conversation history using the boundary the user actually heard, then continue answering from the new city.

The concept most easily substituted here is "cancelable." Having a cancel() interface only proves that a response can be terminated, not that the system knows when it should terminate, nor that the state has been repaired after termination.

The OpenAI Realtime VAD documentation provides a good boundary sample. The API produces input_audio_buffer.speech_started and speech_stopped events; server_vad mainly segments by silence, and semantic_vad judges completion based on user utterance; in voice conversations, interrupt_response can also be set. These capabilities allow applications to build an interruption chain, but the events and switches themselves do not answer for the application "which interaction intent does this overlap represent."

Inference layer acceptance testing should also not just test "whether the cancel request returns success." At minimum, test: the time new input reaches the model, the time generation stops, the time the client actually stops playing, the time re-planning starts, and whether history correction is complete. The last item is often easier to miss than the previous ones.

4. Interaction Cognition Layer: Detecting Sound Does Not Equal Understanding Floor Control

When a new sound appears while the model is speaking, it can correspond to completely different actions.

• "Uh-huh," "right," "I'm listening" may be backchannels, usually not requiring the agent to immediately yield the floor.
• "Wait," "no," "stop for a moment" are usually explicit takeovers, requiring a quick stop and waiting for new instructions.
• A user's inhalation, filler words, or half-sentence repetition while thinking may mean they are still organizing their expression.
• Bystander conversations, TV, vehicle sounds, and speaker echo should not automatically create a new user turn.
• When a user corrects an entity, the old answer needs to be stopped, the task goal retained, and local conditions replaced.

Traditional VAD mainly answers speech / no-speech. Even with semantic end-of-turn detection added, it primarily helps judge whether a segment of speech is complete and does not naturally cover backchannels, third-party speech, background noise, and partial corrections. These scenarios require combining acoustic evidence, speaker cues, current task state, keywords, overlap duration, and interaction history for judgment.

One should not prematurely claim that a model "understands interruptions." A more robust engineering expression is: the system makes an action decision for overlapping input with a reason, confidence level, and consequence, and can recover from incorrect decisions.

For example, the system can output:

decision = BACKCHANNEL_ACCEPT
reason = short_ack_without_task_change
action = keep_speaking_and_keep_listening
confidence = 0.82

It can also output:

decision = USER_TAKEOVER
reason = explicit_stop_intent
action = cancel_response_clear_playback_reopen_floor
confidence = 0.97

This does not necessarily require a separate "floor control model." It only requires the system to make the decision object explicit, rather than hiding all logic behind a VAD threshold and several unobservable callbacks.

5. The Real Challenge Is Four States Existing Simultaneously

The three layers of capability ultimately converge into four running states that easily diverge.

The first is the audio capture state: which audio has been collected, uploaded, recognized, and belongs to the primary user or the environment. The second is the model response state: which response_id is being generated, whether it has been canceled, and at what output position the cancellation occurred. The third is the playback state: which audio has entered the device buffer, and which millisecond the user actually heard. The fourth is the conversation history: what the system believes it has already said, and what the next round model will see.

Production incidents are usually not a single component completely failing, but the four states giving different answers to the same event.

Divergence 1: Model Stopped, Speaker Didn't

The server-side cancellation succeeds, but the audio already sent to the client is still in the queue. Logs show the response is canceled, but the user hears half a sentence tail. The fix is not just calling cancel faster, but adding a playback_generation or equivalent version number to the playback queue, so subsequent audio from the old response cannot continue to be enqueued, and allowing local immediate clearing.

Divergence 2: Speaker Stopped, History Not Truncated

The client stops playback in time, but the server keeps the entire assistant text in the history. The next round model will assume the user heard the unplayed part, leading to skipped steps, misaligned references, or repeated confirmations. The system must record heard_until_ms or an equivalent played boundary and correct the history to the true shared context.

Divergence 3: New Sound Detected, But No Stable Event Identity

The same speech segment is interpreted repeatedly by VAD, ASR, gateway, and business layer, potentially triggering a cancellation, creating a new turn, and being submitted a second time after echo cancellation. Without input_event_id, source, and causal chain, the team can only see multiple seemingly correct logs and cannot prove which decision led to the user experience.

This is also why full-duplex systems need an event protocol, not more scattered callbacks.

6. A Minimal Full-Duplex Event Ledger

To make the three layers and four states traceable, one can start with the following fields, without needing to design a massive framework all at once.

input_event_id: in_042
source: primary_user
captured_at_ms: 18420
speech_kind: explicit_interruption
floor_owner_before: assistant
active_response_id: resp_017
decision: user_takeover
decision_reason: explicit_stop_and_task_correction
decision_confidence: 0.97
model_cancel_at_ms: 18492
playback_stop_at_ms: 18541
heard_until_ms: 1270
history_truncated_to: assistant_audio_1270ms
next_action: await_corrected_city

These fields serve four responsibilities.

• input_event_id connects acoustic events, model actions, player actions, and business results.
• active_response_id and playback_generation prevent old responses from being injected after cancellation.
• heard_until_ms brings the true playback boundary back into the history, rather than assuming generated content equals heard content.
• decision_reason and decision_confidence allow false interruptions and missed interruptions to be categorized for review.

The event ledger does not need to be tied to a specific model interface. WebRTC, WebSocket, SIP, end-to-end voice models, or cascaded ASR / LLM / TTS can all use the same causal fields. Different systems only differ in who produces each field and when it is submitted.

7. Four Adversarial Probes, More Valuable Than a "Can Interrupt" Demo

To judge whether a system has truly formed three-layer full-duplex, run at least four adversarial probes.

Probe A: Backchannels Must Not Automatically Seize the Floor

The agent explains a step that takes ten seconds, and the user says "uh-huh" or "right" in the middle. The passing condition is not "the system heard it," but that the system records the backchannel, continues the current response, and does not repeatedly create new turns. If all short sounds trigger a stop, it means the inference layer is cancelable, but the interaction cognition layer has not passed.

Probe B: Entity Correction Must Stop, Clear Queue, and Re-plan

When the agent answers about Hangzhou weather, the user says "not Hangzhou, Qingdao." Record the four time segments from new input arrival to model stop, player stop, history truncation, and new response start. Testing only one of these delays will hide state divergence.

Probe C: Bystander Speech Must Not Contaminate the Main Session

Add bystander conversation or TV sound while the agent is playing. The system should be able to mark it as third-party or low-confidence environmental speech, at minimum not immediately interrupting and writing it into the primary user history. If the system cannot identify the speaker, it should adopt a conservative strategy, such as brief suppression, requesting confirmation, or waiting for a clearer takeover signal.

Probe D: Content the User Didn't Hear Must Not Remain in History

Force an interruption in the middle of a response, then in the next turn ask "what was the last thing you said?" If the model references the canceled and unplayed tail sentence, it means the playback state and history state have not converged.

These four probes cover backchannels, valid interruptions, noise boundaries, and history consistency respectively. They expose production problems better than a smooth "interrupt anytime demo."

8. What Metrics Should Be Looked at for Each Layer

The transport layer can look at RTT, jitter, packet loss, uplink audio breaks, downlink buffering, and device switching recovery. The inference layer can look at input acceptance delay, model cancellation delay, playback stop delay, re-planning delay, and history correction success rate. The interaction cognition layer needs to look at false interruptions, missed interruptions, mistaken interruptions, backchannel retention rate, bystander speech contamination rate, and pause waiting quality.

There is tension between these metrics.

Shortening silence_duration_ms may respond faster but may also increase thinking pauses being misjudged as end-of-turn. Lowering the interruption threshold may shorten valid interruption time but may also increase false stops caused by background sound and backchannels. Making local speaker gating very aggressive may allow users to take over faster but may also turn normal two-person overlap into a harsh cut-off.

Therefore, acceptance testing cannot use a single "full-duplex latency" score. At minimum, speed, classification correctness, state consistency, and task completion must be placed on the same results table.

9. Not All Scenarios Should Pursue Full-Duplex

Continuous interaction adds value but also adds state space.

In open dialogue, language practice, complex customer service, and long-task agents, where users frequently pause, correct themselves, backchannel, or insert new conditions, three-layer full-duplex is likely worth building. For short commands like "turn on the light," "stop," "return to charge," clear turn-taking, strong confirmation, and deterministic execution may be more reliable. Compliance broadcasts, financial confirmations, or high-risk actions may also intentionally prohibit overlap, requiring the key conditions to be fully broadcast first before accepting explicit confirmation.

Simpler alternatives include: maintaining half-duplex but optimizing endpoint detection; only supporting explicit keyword interruption; locally gating the speaker during playback but not letting arbitrary background sound enter the main history; opening continuous interaction for a few high-value scenarios rather than switching the entire product at once.

Choosing these alternatives does not mean the system is behind. Full-duplex is a scenario capability, not a maturity badge.

Conclusion

WebRTC allows both sides to transmit audio simultaneously, but it will not decide for the system who should speak at this moment.

Natural full-duplex requires three layers to hold simultaneously: media can flow bidirectionally, new input can change the output happening now, and the system can interpret the interaction meaning of overlapping sounds. Entering production adds one more: audio capture, model, playback, and history must give consistent answers to the same event.

So when reviewing a voice agent, don't just ask "can it be interrupted?" Follow up with four things: why the interruption happened, which response was stopped, where the user actually heard up to, and what was left in the next round's history.

Being able to answer these four questions is what moves from bidirectional audio to observable, recoverable, and verifiable full-duplex interaction.

References

1. OpenAI: Introducing GPT-Live, first-party explanation of continuous interaction and action decision-making, accessed 2026-08-08.
2. OpenAI Developers: Voice activity detection, first-party explanation of VAD events, server/semantic VAD, and interrupt_response, accessed 2026-08-08.
3. W3C: WebRTC: Real-Time Communication in Browsers, standard definition of RTCRtpTransceiverDirection and sendrecv, accessed 2026-08-08.
4. Full-Duplex-Bench, research on pause, backchannel, turn-taking, and interruption evaluation.
5. Full-Duplex-Bench v1.5, research on user interruption, listener backchannel, bystander conversation, environmental speech, and stop delay evaluation.

Facts, Derivations, and Unknowns

• Confirmed facts: W3C sendrecv defines media sending and receiving; OpenAI Realtime API provides VAD events and configurable interruption behavior; GPT-Live officially describes continuously processing input during output and frequently choosing interaction actions.
• Author's engineering derivations: The three-layer full-duplex model, four running states, event ledger fields, and four adversarial probes are a systematic analysis framework formed based on public capabilities and production failure modes.
• Unknowns: Whether GPT-Live internally uses an independent floor controller, what kind of dual-stream structure, and how it maintains playback boundaries and history repair are not disclosed in public materials, and this article makes no definitive description.