Evaluating Prompt Scope and Demonstration Similarity in Local LLM Machine Translation
Quick Answer
This study evaluates local LLMs like llama3.2:3b and mistral:latest for machine translation across nine EU languages, revealing that dedicated MT systems outperform LLMs, particularly for Germanic languages.
Quick Take
Few-shot prompting benefits some models but not others, while family-scope prompts expose weaknesses in smaller models.
Key Points
- Evaluated English-to-Romance and English-to-Germanic translations using FLORES devtest.
- Compared three local against OPUS-MT and NLLB-200 baselines.
- Few-shot prompting improved performance for mistral:latest and qwen2.5:14b.
- Family-scope prompting is feasible for stronger LLMs but reveals issues in smaller models.
- Results emphasize the need for diverse evaluation metrics in LLM translation.
DeepSignal Analysis
What happened
This study assesses local large language models (LLMs) for machine translation across nine EU languages, comparing them to dedicated machine translation (MT) systems. It evaluates various prompting strategies, including few-shot and family-scope prompts, revealing that dedicated MT systems generally outperform LLMs, particularly for Germanic languages.
Key evidence
- The study evaluates English-to-Romance and English-to-Germanic translation using the FLORES devtest split for nine EU languages.
- Three local instruction-tuned LLMs, llama3.2:3b, mistral:latest, and qwen2.5:14b, are compared against dedicated MT systems from OPUS-MT and NLLB-200.
- Results indicate that few-shot prompting benefits some models like mistral:latest but negatively impacts llama3.2:3b, while family-scope prompts expose weaknesses in smaller models.
Why it matters
Understanding the performance of local LLMs in machine translation is crucial as these models are increasingly used in practical applications. The findings highlight the limitations of LLMs compared to dedicated MT systems, particularly in specific language families. This research also emphasizes the importance of evaluating translation models based on prompt strategies and their effectiveness across different languages.
Paper Resources
Source Excerpt
(LLMs) are increasingly used as general-purpose translation systems, but their behavior is usually evaluated under a single prompt shape: translate one source sentence into one target language. In practice, users may ask for one target language, for several related languages at once, or for translations conditioned on examples. This paper studies prompt scope and demonstration selection as experimental variables for local LLM machine translation. We evaluate English-to-Roma
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.