Start with the interaction
A voice experience begins with a task: answer a question, help a person navigate a process, or complete an authorized action. The architecture should follow that task. Before selecting a model, define the expected inputs, the business information required, and what a successful exchange sounds like.
A modular pipeline makes the stages explicit
A common design connects automatic speech recognition, a text reasoning model, and speech synthesis. Each stage has an observable boundary: a transcript, a response, and generated audio. This can make targeted tuning and component replacement easier. Errors can also travel between stages, so inspect the entire conversation rather than scoring each component in isolation.
Native audio changes the information available
Audio-native models can work more directly with speech, creating opportunities to use timing and vocal expression alongside the words. Multimodal models can also accept visual context. These capabilities vary by model and need to be tested against your actual use case, including tool integration and output controls.
Streaming and full-duplex are separate decisions
A modular system can stream, and an audio-native model still needs an application that handles interruptions well. Decide how playback stops, what happens to a pending tool call, and how the conversation context changes when the user corrects a request.
Compare on the same conversations
Use representative recordings and scripted interactions for both options. Review critical names and numbers, first audible responses, task correctness, interruption handling, and cost under expected usage. Choose the approach that performs well for the workflow and exposes the controls your team needs.
Further reading
External publications describe their authors’ work. Reference architectures and examples are conceptual.