Rethinking Speech-LLM Integration for ASR: Effective Joint Speech-Text Training by Interleaving
Quick Answer
The proposed Joint Speech-Text Interleaved Pretraining (JSTIP) enhances ASR performance by interleaving speech-text sequences, achieving improved entity accuracy on 38k hours of data.
Quick Take
JSTIP matches domain transcription performance while simplifying adaptation, outperforming traditional ASR and joint training methods, particularly in medical entity recognition.
Key Points
- JSTIP constructs interleaved speech-text sequences for improved ASR training.
- Achieves consistent entity accuracy improvements over ASR-only and joint training baselines.
- Competes with open-source ASR systems in medical entity recognition tasks.
- Zero-shot speech question answering shows reduced modality gaps with JSTIP.
- Utilizes 38k hours of ASR data for robust training results.
Paper Resources
📖 Reader Mode
~2 min readAuthors:Ruchao Fan, Yiming Wang, Rui Zhao, Liliang Ren, Keqi Deng, Xiaoyang Chen, Ali Zare, Bo Ren, Yuxuan Hu, Junkun Chen, Yan Huang, Yelong Shen, Jinyu Li
Abstract:Speech-LLM integration has shown promising results by leveraging extensive textual pretraining, yet its specific benefits for automatic speech recognition (ASR) remain unclear. We observe that as supervised ASR training data increases, the contribution of LLM priors becomes less evident, and simple speech-text joint training under-utilizes textual knowledge. We therefore propose Joint Speech-Text Interleaved Pretraining (JSTIP), an ASR-oriented pretraining strategy that constructs word-level and segment-level interleaved speech-text sequences within aligned pairs for speech-LLM architectures that accept continuous inputs. Experiments on 38k hours of ASR data show consistent entity accuracy improvement compared to ASR-only and joint speech-text training baselines. JSTIP achieves on-par entity recognition performance using domain transcription text compared to synthetic speech-text pairs, simplifying domain adaptation. Benefiting from textual pretraining and domain text data, JSTIP is competitive with open-source ASR and Speech-LLM systems in medical entity recognition. The zero-shot speech question answering behaviors further suggest that interleaving reduces the speech-text modality gap and preserves the LLM generative prior, which is likely the reason for the entity improvements on the ASR task.
| Subjects: | Computation and Language (cs.CL); Audio and Speech Processing (eess.AS) |
| Cite as: | arXiv:2607.01733 [cs.CL] |
| (or arXiv:2607.01733v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.01733 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Ruchao Fan [view email]
[v1]
Thu, 2 Jul 2026 05:42:01 UTC (183 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.