IdiomX A Multilingual Benchmark for Idiom Understanding, Retrieval, and Interpretation
Quick Take
IdiomX introduces a large-scale multilingual benchmark for idiom understanding, featuring over 190K examples across 12K+ idioms. It enhances idiom detection and retrieval through contextual transformer models and hybrid architectures, establishing a new standard for semantic interpretation in idiomatic expressions.
Key Points
- IdiomX dataset includes over 190K contextualized examples and 12K+ idioms.
- Benchmark tasks cover idiom detection, retrieval, and interpretation across languages.
- Contextual transformer models significantly enhance idiom detection performance.
- Hybrid retrieval architectures improve both monolingual and cross-lingual idiom retrieval.
- Idiom interpretation is modeled as a semantic retrieval task, adding interpretability.
Article Content
From source RSS / original summaryarXiv:2606. 02584v1 Announce Type: new Abstract: Idiomatic expressions remain a persistent challenge for natural language processing because their meanings are often non-compositional, context-dependent, and difficult to align across languages. Existing idiom resources are often limited in scale, contextual diversity, or multilingual coverage, restricting their utility for modern language models.
We introduce IdiomX, a large-scale multilingual benchmark for idiom understanding, retrieval, and interpretation, constructed through a reproducible multi-stage pipeline combining lexical resource extraction, large-scale normalization, controlled large language model enrichment, and structured validation.
The resulting dataset contains over 190K contextualized examples spanning 12K+ idioms, with aligned English, Arabic, and French semantic representations, idiomatic and literal usage labels, and rich linguistic metadata. Building on this resource, we define a unified four-task benchmark covering idiom detection, context-to-idiom retrieval, Arabic-to-English idiom retrieval, and idiom interpretation, extending evaluation from figurative recognition to semantic grounding and explainable meaning retrieval.
Experiments show that contextual transformer models substantially improve idiom detection, while hybrid retrieval and reranking architectures significantly strengthen both monolingual and cross-lingual idiom retrieval. Results further demonstrate that idiom interpretation can be effectively modeled as a semantic retrieval task, introducing interpretability as a complementary benchmark dimension.
Overall, IdiomX provides a scalable benchmark for studying idiomatic language as a progression from detection to retrieval and semantic interpretation, and offers a modular framework extensible to additional languages and figurative reasoning tasks
Reader Mode unavailable (could not extract clean content).
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents?
The REFLECT benchmark reveals that current LLM judges are unreliable, achieving below 55% accuracy in evaluating reasoning and evidence use, highlighting the need for improved evaluation methods for deep research agents.