Indi-RomCoM: Code-Mixed Benchmark for Evaluating LLMs on Romanized Indic-English Instructions
Quick Answer
The Indi-RomCoM benchmark evaluates LLMs on Romanized Code-Mixed instructions, revealing significant performance drops, especially as code-mixing density increases.
Quick Take
, including proprietary and open-weight models, consistently struggle with RCM tasks, highlighting the need for improved multilingual systems.
Key Points
- Indi-RomCoM spans seven instruction tasks across four Indic languages.
- LLMs show consistent underperformance on RCM instructions in zero- and few-shot settings.
- Performance degradation increases with higher code-mixing density.
- Reasoning tasks perform better than detection tasks like Toxicity.
- The benchmark aims to aid in developing inclusive multilingual AI systems.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Romanized Code Mixing (RCM), where bilingual speakers fluidly blend local languages with English in Roman script, has emerged as the dominant form of communication across multilingual communities. While Large Language Models (LLMs) perform strongly on monolingual and native-script benchmarks, their ability to follow instructions and reason over RCM-based content remains largely unexplored. To this end, we introduce the Indi-RomCoM benchmark for facilitating systematic evaluation on Indic Romanized Code-Mixed instructions. Our benchmark spans seven instruction-following tasks, four widely spoken Indic languages, and three controlled code-mixing intensity levels. We extensively evaluate a suite of LLMs covering proprietary, open-weight, and Indic-focused models under zero- and few-shot settings. LLMs consistently underperform on RCM instructions, with performance degrading as code-mixing density increases. Furthermore, reasoning tasks suffer less degradation than detection tasks (e.g., Toxicity) because the generated explanations offer necessary context. We believe Indi-RomCoM helps the community in developing inclusive multilingual systems.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2606.30790 [cs.CL] |
| (or arXiv:2606.30790v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2606.30790 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Avisha Das [view email]
[v1]
Mon, 29 Jun 2026 18:19:24 UTC (6,494 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.