Jointly Improving Dialect Identification and ASR in Indian Languages using Multimodal Feature Fusion
Quick Answer
This paper shows that A multimodal framework combining Bottleneck Encoder and RoBERTa improves Automatic Speech Recognition (ASR) and Dialect Identification (DID) for eight Indian languages, achieving an average DID accuracy of 81.63%, with a Character Error Rate (CER) of 4.65% and Word Error Rate (WER) of 17.73%.
Quick Take
This approach addresses the challenges of low-resource languages and their dialectal variations effectively.
Key Points
- Proposed a multimodal framework for joint ASR and DID improvement.
- Utilized Bottleneck Encoder for dialectal feature extraction.
- Achieved 81.63% average DID accuracy across 33 dialects.
- Character Error Rate (CER) of 4.65% and Word Error Rate (WER) of 17.73%.
- Addresses challenges of low-resource Indian languages effectively.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Automatic Speech Recognition (ASR) and Dialect Identification (DID) are crucial for Indian languages, many of which are low-resource and exhibit significant dialectal differences. Existing methods often optimize ASR or DID individually, resulting in performance trade-offs. In this work, we propose a multimodal framework that jointly improves ASR and DID. Our method employs a Bottleneck Encoder to extract dialectal features from Conformer-based speech representations and a RoBERTa encoder to process ASR-generated CTC embeddings. A gating mechanism merges these features, followed by an attention encoder to refine the representations. The learned embeddings are concatenated with Conformer outputs to enhance ASR features. Evaluated on eight Indian languages with thirty-three dialects, our method achieves an average DID accuracy of 81.63% and average CER and WER of 4.65% and 17.73%, respectively. These results highlight the effectiveness of our method for joint ASR-DID modeling.
| Subjects: | Computation and Language (cs.CL); Audio and Speech Processing (eess.AS) |
| Cite as: | arXiv:2607.02862 [cs.CL] |
| (or arXiv:2607.02862v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.02862 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Prasanta Ghosh Prof. [view email]
[v1]
Fri, 3 Jul 2026 01:53:05 UTC (220 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.