Structured Pruning of Large Language Models via Power Transformation and Sign-Preserving Score Aggregation with Adaptive Feature Retention
Quick Answer
This paper presents a novel structured pruning method for large language models, addressing challenges in Adaptive Feature Retention (AFR) by using power transformation and sign-preserving score aggregation.
Quick Take
Experiments on Llama-3-8B, Vicuna-v1.5-13B, and LLaVA-v1.5-13B show that the method maintains accuracy while enhancing inference speed, making it a practical solution for optimizing .
Key Points
- Introduces a unified approach to structured pruning of large language models.
- Addresses distribution mismatch, sign information loss, and outlier influence.
- Maintains accuracy comparable to unstructured pruning methods.
- Achieves practical inference speedup on multiple LLMs.
- Tested on Llama-3-8B, Vicuna-v1.5-13B, and LLaVA-v1.5-13B.
Paper Resources
📖 Reader Mode
~2 min readAbstract:This paper proposes an improved structured pruning method for large language models (LLMs) that addresses key challenges in adapting Adaptive Feature Retention (AFR), an unstructured pruning technique, to structured pruning. When applying AFR to structured pruning, three major problems arise: distribution mismatch between heterogeneous pruning scores, loss of sign information indicating optimization direction consistency, and influence of outliers. To address these issues, we propose a unified approach combining power transformation for nonlinear distribution alignment, sign-preserving score aggregation, and percentile-based outlier removal. Experiments on Llama-3-8B, Vicuna-v1.5-13B, and LLaVA-v1.5-13B demonstrate that our method maintains accuracy comparable to unstructured pruning while achieving practical inference speedup through structured pruning.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.08027 [cs.CL] |
| (or arXiv:2607.08027v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.08027 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yasunori Ishii Mr [view email]
[v1]
Thu, 9 Jul 2026 01:05:42 UTC (1,064 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.