Mind the Cap: Output-Budget Regimes Change the Measured Multilingual Reasoning Gap
Quick Answer
The study reveals that the multilingual reasoning gap in models like Qwen3-8B and Llama-3.1-8B-Instruct is significantly influenced by output-token caps, with variations up to 57 points across different budgets.
Quick Take
Length normalization can shift accuracy scores by 38.9 points, indicating that the cap should be treated as an independent variable for accurate multilingual evaluation.
Key Points
- Measured multilingual reasoning gaps can swing by up to 57 points across token budgets.
- Length normalization impacts accuracy scores by up to 38.9 points under tight caps.
- A Thai vocabulary extension closed a 4.9-point gap at a specific budget.
- Native accuracy saturation occurs at a budget cap of 1024 tokens.
- Accuracy is influenced by the announced budget, not just the enforced cap.
DeepSignal Analysis
What happened
The study investigates the impact of output-token caps on multilingual reasoning in models like Qwen3-8B and Llama-3.1-8B-Instruct. It finds that the measured reasoning gap can vary significantly based on the token budget, with differences up to 57 points. Length normalization also affects accuracy scores, indicating the need to treat the output cap as an independent variable.
Key evidence
- The multilingual reasoning gap for Qwen3-8B and Llama-3.1-8B-Instruct varies by up to 57 points across different output-token budgets.
- Length normalization can shift accuracy scores by as much as 38.9 points, suggesting that the output cap influences evaluation outcomes.
- The study evaluated models on 540,000 independently hard-capped decodes, rejecting every null hypothesis in a series of Holm-corrected tests.
Why it matters
Understanding how output-token caps affect multilingual reasoning is crucial for accurate model evaluation. The findings suggest that current evaluation practices may overlook significant variations in performance due to these caps. By treating the output cap as an independent variable, researchers can better assess model capabilities across different languages and contexts, leading to more reliable benchmarks.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Multilingual evaluations report accuracy at a single output-token cap, but languages need different numbers of tokens to express the same content, so the cap is a hidden experimental variable. We test whether the native-vs-translate gap on MGSM (German, Thai, Swahili) is a token-budget artifact for Qwen3-8B and Llama-3.1-8B-Instruct under four prompting strategies. The measured gap swings by up to 57 points across budgets, length normalization moves it by up to 38.9 points where the cap binds, and at tight caps normalization can reverse which strategy scores higher. We prospectively froze the sweep's three Qwen peaks and its near-zero value at 1024 and evaluated them on 540,000 independently hard-capped decodes: a second frozen family of six Holm-corrected tests rejects every null. The frozen test at $B^*=1024$ still fails to reject because native accuracy has already saturated there; above saturation, the residual difference is a strategy-performance gap, not an identified reasoning deficit. The same truncation channel prices a cost-ordered adaptation ladder: a cross-fitted Thai vocabulary extension closes 0.0 points of the gap at the frozen budget and 4.9 points where 19% of traces still truncate. A third frozen family varies only the announced budget at a fixed enforced cap; announcing 128 rather than 2048 tokens moves Thai native accuracy by 5.1 points, so accuracy is not a function of the enforced cap alone. A correct-emission timing identity computed from one long-cap run matches the three pre-specified MGSM peaks to 0.65 points and, in an exploratory Qwen-only analysis of three further benchmarks, tracks held-out items to 0.92 points, locating the peak exactly in five of seven cells. Treat the output cap as an independent variable and report accuracy across the budget regime, not at a single budget.
| Comments: | 15 pages, 2 figures, 11 tables. Under review |
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| ACM classes: | I.2.7; I.2.6 |
| Cite as: | arXiv:2608.04160 [cs.CL] |
| (or arXiv:2608.04160v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2608.04160 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Ankit Goyal [view email]
[v1]
Tue, 4 Aug 2026 19:18:41 UTC (259 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.