Indi-RomCoM: Code-Mixed Benchmark for Evaluating LLMs on Romanized Indic-English Instructions
Quick Answer
The Indi-RomCoM benchmark evaluates LLMs on Romanized Code-Mixed instructions, revealing significant performance drops, especially as code-mixing density increases.
Quick Take
The Indi-RomCoM benchmark evaluates LLMs on Romanized Code-Mixed instructions, revealing significant performance drops, especially as code-mixing density increases. LLMs, including proprietary and open-weight models, consistently struggle with RCM tasks, highlighting the need for improved multilingual systems.
Key Points
- Indi-RomCoM spans seven instruction tasks across four Indic languages.
- LLMs show consistent underperformance on RCM instructions in zero- and few-shot settings.
- Performance degradation increases with higher code-mixing density.
- Reasoning tasks perform better than detection tasks like Toxicity.
- The benchmark aims to aid in developing inclusive multilingual AI systems.
Paper Resources
Article Content
From source RSS / original summaryarXiv:2606. 30790v1 Announce Type: new Abstract: Romanized Code Mixing (RCM), where bilingual speakers fluidly blend local languages with English in Roman script, has emerged as the dominant form of communication across multilingual communities. While Large Language Models (LLMs) perform strongly on monolingual and native-script benchmarks, their ability to follow instructions and reason over RCM-based content remains largely unexplored.
To this end, we introduce the Indi-RomCoM benchmark for facilitating systematic evaluation on Indic Romanized Code-Mixed instructions. Our benchmark spans seven instruction-following tasks, four widely spoken Indic languages, and three controlled code-mixing intensity levels. We extensively evaluate a suite of LLMs covering proprietary, open-weight, and Indic-focused models under zero- and few-shot settings.
LLMs consistently underperform on RCM instructions, with performance degrading as code-mixing density increases. Furthermore, reasoning tasks suffer less degradation than detection tasks (e. g. , Toxicity) because the generated explanations offer necessary context. We believe Indi-RomCoM helps the community in developing inclusive multilingual systems.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Quantifying Prior Dominance in Systems
The study introduces the Normalized Context Utilization (NCU) metric to evaluate Retrieval-Augmented Generation (RAG) systems, revealing that Small Language Models (SLMs) outperform larger models in factual extraction. The findings indicate that traditional scaling laws yield diminishing returns, with a commercial API frequently failing against adversarial evidence due to systemic confidence collapse.