Are you speaking my languages? On spoken language adherence in multimodal LLMs
Quick Answer
This study addresses language adherence issues in LLM-based ASR systems, proposing a soft prompting method to enhance multilingual transcription accuracy.
Quick Take
Three strategies—zero-shot prompting, supervised fine-tuning, and Chain-of-Thought reasoning—are evaluated for their effectiveness in reducing language violations while maintaining ASR performance across multiple languages.
Key Points
- Proposes a soft prompting approach to enhance multilingual ASR performance.
- Introduces a novel metric to quantify language adherence violations.
- Evaluates zero-shot prompting, supervised fine-tuning, and CoT reasoning.
- Finds trade-offs in strategy selection based on compute constraints.
- Demonstrates effectiveness in reducing language violations across languages.
Paper Resources
Source Excerpt
arXiv:2606. 17281v1 Announce Type: new Abstract: While (LLM) based Automatic Speech Recognition (ASR) enables seamless multilingual use, models often misidentify the output language, compromising transcription fidelity and downstream application quality. To preserve flexibility and code-switching capabilities, we propose a soft prompting approach that hints at potential spoken languages without strictly constraining the output. …
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.