Building a European Multilingual Evaluation Dataset: The MMLU Localisation Project within the EMT Network
Quick Answer
This paper shows that The MMLU Localisation Project, a collaboration between the DGT and EMT, aims to translate the MMLU dataset into 11 European languages, enhancing LLM evaluation benchmarks while providing master's students with practical training in translation and project management.
Key Points
- Project localizes dataset into 11 European languages for better evaluation.
- Offers master's students hands-on training in translation and project management.
- Highlights challenges in methodology, administration, and workflow during localization.
- Collaboration involves the Directorate-General for Translation and EMT network.
- Results will be presented at the 2026 International Conference on Translation Technology.
Paper Resources
📖 Reader Mode
~2 min readAbstract:This paper reports on a collaboration between the Directorate-General for Translation (DGT) and the European Master's in Translation (EMT) to localise the MMLU dataset into 11 European languages. Beyond creating a more inclusive benchmark for LLM evaluation, the project offers master's students authentic, project-based professional training in translation, revision, project management, and multilingual coordination, while highlighting key methodological, administrative, and workflow challenges.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2607.18432 [cs.CL] |
| (or arXiv:2607.18432v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.18432 arXiv-issued DOI via DataCite (pending registration) |
|
| Journal reference: | Proceedings of the 3rd International Conference on New Trends in Translation and Interpreting Technology, June 24 - 27, 2026, Dubrovnik, Croatia |
| Related DOI: | https://doi.org/10.26615/issn.2815-4711.2026_007
DOI(s) linking to related resources |
Submission history
From: Susana Valdez [view email]
[v1]
Mon, 20 Jul 2026 18:33:59 UTC (456 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.