Follow one request through the system
Imagine asking an assistant to move a delivery to Friday. The system must capture the sound, identify the request, check what changes are possible, and communicate the confirmed result. Each step can succeed or fail independently. Understanding the path helps explain why a pleasant voice can still produce a poor assistant.
A common architecture uses speech-to-text, a language model, and text-to-speech. These are often abbreviated STT, LLM, and TTS. Other architectures handle audio input and output through an integrated model. The modular pipeline remains a useful way to explain responsibilities, without implying every voice system contains three separate services.
Recognition estimates what was said
A recognizer converts audio into text. It must cope with accents, background sound, names, numbers, and corrections. Streaming recognition can provide provisional results while the caller is still speaking; later updates may change those words. [1]
Recognition quality affects downstream interpretation. If “Friday” becomes “Thursday,” later stages may confidently work on the wrong date. Track the fields that matter for the task as well as overall transcription quality. For important details, a short confirmation can expose an error before it becomes an action.
Response generation decides what to say or request
The language model receives the recognized text and relevant conversation state. It can draft a response or propose a structured operation, such as checking delivery availability. Application code must validate and execute that operation and return its actual outcome.
Generated language is not evidence that the external system changed. Keep the reply consistent with confirmed results, and distinguish unavailable, failed, and uncertain outcomes. Maintain useful context across turns, including corrections and unfinished work, rather than treating every utterance as an unrelated request.
Synthesis turns response text into speech
Text-to-speech produces audio from the response. Streaming synthesis can begin before a long response is fully generated, depending on the input and model requirements. Enough phrase context is still needed for intelligible pronunciation, especially for numbers, abbreviations, and ambiguous text.
The audio must then be delivered, buffered, and played. A first audio byte at the server is not the same event as the first audible word. Capture and playback behavior belong in both testing and performance measurement.
The runtime coordinates the conversation
Turn handling decides when it is appropriate to respond. Interruption handling decides what to stop when the caller speaks during output. Session management tracks active work, handles disconnects, and rejects obsolete output after a correction.
Test these responsibilities together. Ask for Friday, interrupt with “actually Monday,” and verify that the right date reaches the business system and the spoken confirmation. A component benchmark cannot establish that outcome. The complete agent is the combination of models, transport, state, and recovery behavior that carries a spoken request through to a useful result.
Research and references
- [1] Evaluating incremental speech recognition for conversational AI ↗
- [2] Research on a speech-text foundation model for real-time dialogue ↗
- [3] RFC 8834: Media Transport and Use of RTP in WebRTC ↗
Technical review: November 9, 2026. Research supports the underlying concepts; implementation recommendations and illustrative scenarios are editorial guidance, not measured product results.