People Making Conversation: Evaluating How Naturally Speech Models Converse
Quick Answer
This study evaluates real-time conversational speech models, finding that hosted APIs like GPT Realtime and Gemini 3.1 Flash Live respond at almost every pause, with latencies of 1-4 seconds.
Quick Take
Full-duplex models like PersonaPlex and Moshi are unique in their ability to remain silent and interrupt within human reaction times. Overall, hosted systems recall 41-60% of conversation facts when provided with text history.
Key Points
- Evaluated 427 pauses and 482 turn ends from 176 unscripted conversations.
- Hosted APIs respond almost every time, with latencies of 1-4 seconds.
- Full-duplex models are the only ones that frequently remain silent.
- Hosted systems recall 41-60% of conversation facts based on text history.
- Study results will be shared, including labels and outputs.
DeepSignal Analysis
What happened
A study evaluated six real-time conversational speech models across various metrics. Hosted APIs like GPT Realtime and Gemini 3.1 Flash Live responded at nearly every pause with latencies of 1-4 seconds. In contrast, full-duplex models such as PersonaPlex and Moshi demonstrated the ability to remain silent and interrupt within human reaction times, a unique feature among the systems tested.
Key evidence
- The study involved 427 pauses and 482 turn ends from 176 unscripted conversations, assessing how models responded at these points.
- Hosted systems recalled 41-60% of conversation facts when given text history, while full-duplex models recalled 17-22% when hearing audio.
- Latency for hosted APIs was about 1-4 seconds with silence-based endpointers, while semantic turn detection resulted in about 8 seconds of latency.
Why it matters
Understanding the performance of these conversational models is crucial for their application in real-time interactions. The ability to respond quickly and accurately can significantly influence user experience in various domains, from customer service to personal assistants. The findings highlight the strengths and weaknesses of different models, informing future developments in conversational AI.
Paper Resources
📖 Reader Mode
~2 min readAbstract:A conversational speech model has to decide, at every pause, whether to speak. Existing evaluations test boundary detection, policy adherence, or aggregate statistics, not this decision. We place real-time speech systems at 427 pauses and 482 turn ends from 176 unscripted conversations in Seamless Interaction, give each the conversation so far, and score what it generates against what the person did: whether and how soon it speaks at a pause; its latency, length, and wording at a turn end; and whether it can answer a question about the preceding 30 to 120 seconds. Across six systems, the hosted APIs (GPT Realtime, Gemini 3.1 Flash Live, Grok Voice) respond at almost every pause whichever endpointer they use; only their latency, about 1 to 4 s with silence-based endpointers and about 8 s with semantic turn detection, decides whether the words land inside the pause. The full-duplex models (PersonaPlex, Moshi) are the only systems that regularly stay silent and the only ones that interrupt within a human reaction time. Hosted systems reply longer than the person. Given the history as text, hosted systems recall 41 to 60% of conversation facts under a Claude Opus 5.5 judge and 19 to 26% under a strict word matcher; hearing it as audio, full-duplex models recall 17 to 22% and 12 to 15%. We release the labels, points, harness, and every output.
| Comments: | Accepted at the NeurIPS 2026 Workshop on Real-Time Conversational Agents |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.12802 [cs.CL] |
| (or arXiv:2610.12802v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.12802 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Iman Modarressi [view email]
[v1]
Thu, 8 Oct 2026 23:18:20 UTC (1,772 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →The "10th Juror": Open-Set Standpoint Screening for Bureaucratic Bias Detection
MARS-Gov introduces a framework for detecting bureaucratic bias in Dutch government documents, achieving a new state-of-the-art F1 score of 0.880. This model outperforms existing zero-shot detectors by 20.2 points and reduces unnecessary interventions to just 2.5%. The framework's dynamic '10th juror' adapts to emerging biases, enhancing legal language processing.