Building voice AI that survives production

It is easier than ever to build a voice AI demo. Wire a speech-to-text API to an LLM to a text-to-speech voice, and within an afternoon you have something that talks back. It is also easier than ever to mistake that demo for a system. The gap between the two is where most of the real engineering lives, and it is the gap we are usually hired to close.

Latency is the whole product

A voice conversation has a tempo. Cross a few hundred milliseconds of silence and the interaction stops feeling like a conversation and starts feeling like a phone tree. That single constraint shapes the entire architecture.

You cannot wait for a full LLM response before you start speaking. The response has to stream, and the moment a complete sentence is available it should go to text-to-speech and start playing while the rest is still generating. You cannot wait for a clean end-of-speech signal before you start transcribing, either — interim transcripts and voice-activity detection let you act on intent before the user has finished the sentence. Every stage is a stream feeding the next, and the time-to-first-spoken-token is the number that actually matters.

Interruptions are not an edge case

In real conversations, people interrupt. They cut you off, correct themselves, and talk over the pauses. A voice system that can only operate in strict turns feels broken to anyone who uses it for more than a minute.

That means the speaking state has to be cancellable at any instant. When the user starts talking, whatever the assistant was saying needs to stop, the queued audio needs to clear, and the system needs to start listening again — cleanly, without dropping the thread of the conversation. Designing for interruption from the start is far cheaper than retrofitting it.

The transcript is not the input

Raw transcripts are messy: filler words, false starts, and the occasional misheard phrase. Feeding them straight to your backend couples the quality of your AI to the quirks of your speech model. We treat the cleaned transcript as the real interface — and, crucially, keep the backend unaware of whether the input arrived by voice or by text. The voice layer stays transparent, which means the same reasoning code serves both channels and you only have one system to reason about.

Reliability is a systems problem, not a model problem

When voice AI fails in production, it usually is not the model's fault. It is a dropped websocket, a telephony edge case, an upstream timeout, or simple concurrency the prototype never saw. These are ordinary distributed-systems problems, and they respond to ordinary distributed-systems discipline: backpressure, graceful degradation, explicit failure modes, and enough tracing and structured logging that a bad call is diagnosable afterward instead of a mystery.

None of this is glamorous, and none of it shows up in a demo. It is the difference between something that works once on stage and something an organization can route real calls through every day. That difference is the work — and it is the work worth doing well.