In natural human conversation, comfortable response latency ranges between 150ms and 300ms. Once conversational delay crosses 500ms, the interaction feels sluggish and robotic; beyond 1,000ms, users begin talking over the assistant, resulting in collision errors and broken dialogues.
While engineering real-time conversational agents for IntelliCraft AI, our initial prototype relied on traditional REST polling and base64-encoded audio uploads. Average round-trip latency hovered between 1,800ms and 2,500ms. Here is how we overhauled the architecture using full-duplex WebSockets, chunked PCM binary streaming, and token-pipelined text-to-speech to compress latency down to under 220ms.
The Latency Breakdown: Why HTTP Polling Fails Voice
In a standard HTTP architecture, the client waits until the user finishes speaking, records a complete WAV file, encodes it as base64, and sends an HTTP POST. The server must then decode the file, send it to a Speech-to-Text (STT) model, wait for the full transcription, pass the prompt to an LLM, wait for the entire text completion, send the text to a Text-to-Speech (TTS) engine, and finally return an MP3 file to the client.
Pipelining Audio Streaming: Sub-Second Time-to-First-Sound
The secret to sub-250ms responsiveness is not making individual models 10x faster—it is pipelining operations concurrently so that downstream tasks begin processing before upstream tasks have finished.
As the user speaks into the microphone, the browser's AudioWorklet slices PCM audio into 20ms raw binary frames transmitted over a persistent WebSocket connection. The Speech-to-Text engine streams partial transcripts in real time. The moment punctuation or a complete semantic clause is detected, the LLM begins streaming tokens immediately.
Handling User Barge-In and Interruption
A critical challenge in conversational voice interfaces is barge-in. If the assistant is speaking and the user interrupts with "Wait, cancel that!", a naive system keeps playing the old synthesized audio while processing the new request.
Our client-side AudioWorklet incorporates a lightweight Voice Activity Detector (VAD). The millisecond user voice energy crosses threshold while audio is playing, the client sends a binary INTERRUPT signal across the WebSocket. The server instantly aborts LLM generation, clears downstream TTS queues, and drops active audio streams, making the assistant stop speaking in under 35 milliseconds.
