Define the clock before reporting a number
“Latency” can describe connection setup, recognition, text generation, or the wait before the caller hears a reply. Those are different measurements. For conversational response time, define the starting point as the end of the user’s relevant speech and the ending point as audible response playback on the client.
Also measure the first greeting separately. A cold worker or permission prompt may slow connection setup without affecting later turns. Report which events and clocks were used, especially when a server-side metric is only a proxy for what the listener heard.
Build a timeline for one slow turn
Useful landmarks include audio arrival, speech offset, recognition updates, turn commitment, response generation, tool execution, synthesis output, and playback start. Correlate them with session and turn identifiers. Trace Context standardizes how distributed requests can be linked; your application still needs to instrument its audio events. [1]
Use monotonic clocks for durations within a process. Across devices, account for clock alignment and uncertainty. Subtracting unrelated wall-clock timestamps can produce a precise-looking number that is wrong. When playback cannot be instrumented, label the measurement as server-to-first-audio rather than end-to-end listening latency.
Find the work that actually delays the reply
Streaming stages overlap. Recognition may already be underway before the user finishes, and synthesis may start before the response is complete. Total response time is therefore not generally the sum of full stage durations. It is also not simply the duration of the slowest stage.
Inspect the dependency chain leading to the first useful audio. For an illustrative appointment query, that chain might include a turn decision, an availability lookup, a response phrase, synthesis, and client buffering. A faster recognizer will help little if the required lookup dominates that path.
Optimize with conversational constraints
Test changes to buffering, model selection, placement, and unnecessary serial work one at a time. Independent read operations may overlap; dependent operations still need their prerequisites. Keep deadlines on slow work and communicate progress truthfully when a result is pending.
Earlier turn commitment can reduce waiting while increasing premature responses. Speculative generation can prepare a reply before the user is done, but revised input can invalidate that work. Keep speculative content and side effects separate from committed output, and include regeneration cost in the comparison.
Look beyond the median
Review response-time distributions by ordinary turns, tool turns, startup, device, region, and load. A global median can hide a small group experiencing repeated long waits. Include dropped sessions and failed turns so a change does not appear faster merely because its slowest calls disappear from the sample.
Pair timing with interruption errors, completion rate, and listener review. Save representative slow traces before changing the system, then compare equivalent cases afterward. An improvement is useful when it shortens the wait people experience while preserving correct, understandable conversation.
Research and references
- [1] W3C Trace Context: correlating requests across services ↗
- [2] RFC 8836: Congestion Control Requirements for Interactive Real-Time Media ↗
- [3] Incremental Processing and Voice Activity Projection in Spoken Dialogue Systems ↗
Technical review: November 9, 2026. Research supports the underlying concepts; implementation recommendations and illustrative scenarios are editorial guidance, not measured product results.