Direct Preference Optimization for English-Mandarin Code-Switching Speech Recognition in Audio LLMs
Quick Take
Direct Preference Optimization (DPO) significantly enhances English-Mandarin code-switching transcription in Audio LLMs, reducing error rates by up to 89.6% in-distribution. Training on 100K preference pairs, models now preserve language composition instead of translating, addressing systematic failures in multilingual capabilities.
Key Points
- DPO aligns models to preserve mixed-language content during transcription.
- Training involved 100K preference pairs, totaling 570 hours of data.
- Error rates dropped by 89.6% in-distribution and 20.0% out-of-distribution.
- Three Audio LLMs were trained to improve code-switching transcription.
- Findings indicate DPO effectively enhances multilingual transcription behavior.
Article Excerpt
From source RSS / original summaryarXiv:2605. 23975v1 Announce Type: new Abstract: Audio large language models (Audio LLMs) exhibit systematic failures in transcribing code-switching speech despite strong multilingual capabilities. Focusing on English-Mandarin, we identify three failure modes: language omission, translation-instead-of-transcription, and hallucination.
We apply Direct Preference Optimization (DPO) to align models, constructing preference pairs in which chosen responses preserve mixed-language content while rejected responses mimic failure patterns. Training three Audio LLMs on 100K pairs (570 hours), we observe consistent behavioral shifts: models learn to preserve language composition rather than translating when prompted for transcription. This alignment yields MER reductions up to 89. 6% (in-distribution) and 20. 0% (out-of-distribution).
Our findings suggest DPO can effectively elicit correct code-switching transcription behavior from multilingual Audio LLMs.
Reader Mode unavailable (could not extract clean content).
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents?
The reliability of LLM judges for evaluating deep research agents is critically assessed using the REFLECT benchmark.