ComplexityMT: Benchmarking the Interaction Between Text Complexity and Machine Translation
Quick Answer
ComplexityMT introduces a benchmark for assessing the interaction between text complexity and machine translation, revealing that higher CEFR levels correlate with increased translation difficulty.
Quick Take
ComplexityMT introduces a benchmark for assessing the interaction between text complexity and machine translation, revealing that higher CEFR levels correlate with increased translation difficulty. Evaluating models across six languages, findings indicate that machine translation often alters the CEFR level of the target text, impacting multilingual content generation and translation difficulty estimation.
Key Points
- ComplexityMT evaluates translation difficulty using CEFR levels across six languages.
- Higher CEFR levels lead to increased translation difficulty for most languages.
- Machine translation often shifts the CEFR level of the target text compared to the source.
- Findings provide insights for multilingual pedagogical content generation.
- Research impacts practitioners in machine translation difficulty estimation.
Article Excerpt
From source RSS / original summaryarXiv:2606. 05421v1 Announce Type: new Abstract: When a text is translated, does the translation retain the complexity of the original? We introduce ComplexityMT, a new challenge for assessing how text complexity and machine translation interact with and influence each other, using the Common European Framework of Reference for Languages (CEFR) levels as the measure of text complexity.
Across six languages, including Arabic, Dutch, English, French, Hindi, and Russian, we evaluate three open-weight models, one closed model, and a commercial machine translation system on two tasks: i) correlation of CEFR with translation difficulty, and ii) shifts in CEFR levels of the source texts. Our experiments show that higher CEFR levels make texts more difficult to translate, and that machine translation shifts the CEFR level of the target text compared to the original source, for most languages.
These findings provide new insights for researchers and practitioners working on multilingual pedagogical content generation and machine translation difficulty estimation.
Reader Mode unavailable (could not extract clean content).
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents?
The REFLECT benchmark reveals that current LLM judges are unreliable, achieving below 55% accuracy in evaluating reasoning and evidence use, highlighting the need for improved evaluation methods for deep research agents.