Voice Assistant Endpointing & Barge-In Lab (2D)

Pipeline State

Listening

Silence / countdown

0 ms

Smoothed energy

0.00

Turns completed · Barge-ins

0 · 0

VAD & Endpointing

Energy threshold 0.18

Endpoint silence delay 700 ms

Playback speed 1.0×

How it works

Every voice assistant turn runs the same three-stage decision loop over ~40 ms microphone frames:

1. Voice activity detection (VAD)
   E_t = amplitude_t   (short-term frame energy)
   S_t = α·E_t + (1-α)·S_{t-1}   (EMA smoothing, α=0.35)
   raw_t = S_t > threshold
   active_t = raw_t OR hangover > 0   (~240ms hangover
              bridges gaps between syllables)

2. Endpointing (turn-taking)
   while state == user_speaking and !active:
       silence_ms += frame_ms
   if silence_ms >= endpoint_delay_ms:
       fire endpoint → assistant may now respond

3. Barge-in
   while state == assistant_speaking:
       if active continuously >= 200ms and barge-in enabled:
           cancel TTS playback immediately
           return turn to the user

Energy threshold — how loud a frame must be, after smoothing, to count as speech. Too low and background noise trips the detector; too high and quiet speech gets missed.
Endpoint silence delay — how long the user must stay silent before the assistant assumes the turn is over. Short delays feel snappy but cut off mid-sentence pauses; long delays feel slow but never interrupt.
Barge-in — real assistants keep listening while they talk (full-duplex audio) so a user can interrupt; disabling it here shows the alternative half-duplex design where the mic is effectively ignored during TTS playback.
The two lanes in the 2D view are independent audio tracks: the top lane is the live microphone signal the VAD classifies, the bottom lane is the assistant's synthesized speech (thinking pulses, then a full utterance).

This is the same mechanism behind real products — Alexa, Siri, Google Assistant and every LLM voice agent built on Twilio/LiveKit/OpenAI Realtime — where VAD + endpointing + barge-in decide, frame by frame, who is allowed to talk.

Silence / noise floor User voice detected Barge-in frame Assistant thinking Assistant speaking