PEDBench: Can Speech Models Detect Phonetic Errors?
Quick Answer
This paper shows that The introduction of PEDBench reveals that current state-of-the-art speech models struggle to detect phonetic errors, with low precision in identifying substitutions, deletions, and insertions.
Quick Take
Despite fine-tuning on synthetic data, these models still exhibit a significant sim-to-real domain gap. The benchmark includes diverse datasets from children and scripted human actors, aiming to enhance phonetic error detection in educational and clinical settings.
Key Points
- PEDBench is the first benchmark for phonetic error detection, including 2,605 error instances.
- Evaluation of 13 speech models shows they misclassify correctly pronounced phonemes as errors.
- Fine-tuned models outperform commercial baselines on word-level phonetic error detection.
- The dataset includes real-world recordings and synthetic speech to enhance evaluation.
- Public release of code and datasets is planned to improve PED system accuracy.
Paper Resources
📖 Reader Mode
~2 min readAuthors:Zhonghao Shi, Zackary Rackauckas, Jasmine Yip, Savannah Halpern, Simon Campos, Lauren Brace, Emma Yang, Tiantian Feng, Xuan Shi, Shrikanth Narayanan, Teresa Satterfield, Ying Xu
Abstract:Phonetic error detection (PED) aims to identify phoneme-level substitutions, deletions, and insertions and is essential for educational and clinical speech assessment and intervention. Despite recent advances in speech models for phoneme recognition, their ability to reliably detect fine-grained phonetic errors remains unclear because of a lack of high-quality evaluation data. To address this gap, we introduce PEDBench, the first PED benchmark spanning a continuum from natural to controlled speech. It comprises real-world recordings of spoken errors made by children (N=334), scripted error productions by human actors (N=1,188), and synthetic error productions generated using phone-conditioned text-to-speech (N=1,083). All phonetic annotations were transcribed and verified by two phonetically trained annotators. Our evaluation of 13 state-of-the-art speech models shows that they fail to reliably detect phonetic errors, even under controlled conditions with acted and synthetic speech, where acoustic variability is reduced. The models also exhibit low precision, frequently misclassifying correctly pronounced phonemes as errors. Our speech-language models, fine-tuned on synthetic data, outperform commercial and open-source baselines on word-level PED, although a sim-to-real domain gap remains. We plan to publicly release our code and recipes, along with our acted and synthetic datasets, to support the development of more accurate PED systems.
| Comments: | 21 pages, 10 figures |
| Subjects: | Computation and Language (cs.CL); Sound (cs.SD) |
| Cite as: | arXiv:2610.20830 [cs.CL] |
| (or arXiv:2610.20830v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.20830 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zackary Rackauckas [view email]
[v1]
Wed, 14 Oct 2026 19:15:48 UTC (6,201 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →The "10th Juror": Open-Set Standpoint Screening for Bureaucratic Bias Detection
MARS-Gov introduces a framework for detecting bureaucratic bias in Dutch government documents, achieving a new state-of-the-art F1 score of 0.880. This model outperforms existing zero-shot detectors by 20.2 points and reduces unnecessary interventions to just 2.5%. The framework's dynamic '10th juror' adapts to emerging biases, enhancing legal language processing.