Pruning via Causal Attribution Preserves Reasoning Performance in Large Language Models
Quick Answer
This paper shows that Causal Attribution Pruning (CAP) enhances reasoning performance in large language models like Llama-3 and Mistral-7B, achieving up to 61% accuracy gains over Wanda on ARC-Challenge at 20% sparsity.
Quick Take
CAP identifies critical attention heads based on their causal impact, outperforming traditional pruning methods in preserving performance while reducing inference costs.
Key Points
- CAP estimates performance degradation by masking attention heads during reasoning tasks.
- Achieved relative accuracy gains of up to 61% on ARC-Challenge at 20% sparsity.
- Evaluated on GSM8K, StrategyQA, and ARC-Challenge with Llama-3 and Mistral-7B.
- CAP outperforms magnitude-only and activation-based pruning methods.
- Performance improvements are especially notable at moderate sparsity levels (10-20%).
Paper Resources
Source Excerpt
(LLMs) excel at multi-step reasoning but incur substantial inference cost. We introduce Causal Attribution Pruning (CAP), a training-free method that identifies critical attention heads by measuring their causal impact on reasoning tasks and uses these head-level scores to guide fine-grained weight pruning. For each attention head, CAP estimates the expected performance degradation when the head is masked during forward passes on a small calibration set of reasoning problem
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.