Reinforcement Learning for Data-Efficient Code-Switched ASR
Quick Answer
The study introduces a reinforcement learning approach for adapting audio-language models to code-switched ASR, achieving performance comparable to full dataset fine-tuning with only 10% of the data.
Quick Take
Using Qwen2-Audio, the method effectively reduces translation errors and script contamination, especially for typologically distant language pairs, and demonstrates zero-shot transfer to human-recorded corpora.
Key Points
- Introduces reinforcement learning with verifiable rewards for code-switched ASR adaptation.
- Achieves performance matching full dataset fine-tuning using only 10% of the data.
- Error rate and script fidelity rewards significantly reduce translation errors and contamination.
- Demonstrates effectiveness across 10 language pairs, particularly distant typological pairs.
- Zero-shot transfer to human-recorded code-switching corpus validated.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Audio-language models can be prompted for code-switched speech, but their decoding is not optimized for code-switching and often fails at language boundaries. We propose a practical reinforcement learning with verifiable rewards recipe for data-efficient adaptation of audio-language models to code-switched ASR using group relative policy optimization, combining an error rate reward with a script fidelity reward that penalizes wrong writing systems and a two-pass draft-and-refinement procedure. Using Qwen2-Audio as a reproducible testbed across 10 language pairs, training on only TTS code-switched speech, we show that RLVR with 10% of the data matches LoRA supervised fine-tuning trained on the full dataset, with the largest gains on typologically distant pairs. The error rate reward eliminates translation errors while the script fidelity reward separately reduces script contamination without degradation. These gains transfer zero-shot to a human-recorded code-switching corpus.
| Comments: | Accepted at Interspeech 2026 |
| Subjects: | Computation and Language (cs.CL); Sound (cs.SD) |
| Cite as: | arXiv:2607.02757 [cs.CL] |
| (or arXiv:2607.02757v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.02757 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Peter Vickers [view email]
[v1]
Thu, 2 Jul 2026 20:44:53 UTC (5,114 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.