The Benchmark Illusion: Pruned LLMs Can Pass Multiple Choice but Fail to Answer
Quick Answer
This paper shows that Pruned large language models (LLMs) can excel in multiple-choice tests but fail in open-ended responses, revealing a 'benchmark illusion.' High-sparsity pruning, especially with models like Wanda, leads to recognition errors where correct answers are demoted rather than erased.
Quick Take
This discrepancy suggests that compressed models may not be as reliable as benchmarks indicate, necessitating evaluations based on generative capabilities.
Key Points
- Pruned models perform well in multiple-choice but struggle in open generation tasks.
- High-sparsity pruning leads to recognition errors, particularly in models like Wanda.
- Correct answers may be demoted rather than erased in pruned models.
- Multiple-choice benchmarks may overstate the effectiveness of compressed .
- Evaluations should focus on generative capabilities, not just recognition.
Paper Resources
Source Excerpt
arXiv:2606. 17609v1 Announce Type: new Abstract: Compressing reduces memory use and inference cost, but it can also create failures that standard benchmarks miss. A pruned model may still perform well on multiple-choice evaluations, yet fail to answer the same question in open generation. We ask what pruning changes: does it erase the correct answer, or does it make the answer harder to produce as the top output?
We study this question with multilingual question answering, tracking the same questions before and after pruning. We find a benchmark illusion. …
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.