Structured Pruning of Large Language Models via Power Transformation and Sign-Preserving Score Aggregation with Adaptive Feature Retention
Quick Answer
This paper presents a novel structured pruning method for large language models, addressing challenges in Adaptive Feature Retention (AFR) by using power transformation and sign-preserving score aggregation.
Quick Take
This paper presents a novel structured pruning method for large language models, addressing challenges in Adaptive Feature Retention (AFR) by using power transformation and sign-preserving score aggregation. Experiments on Llama-3-8B, Vicuna-v1.5-13B, and LLaVA-v1.5-13B show that the method maintains accuracy while enhancing inference speed, making it a practical solution for optimizing LLMs.
Key Points
- Introduces a unified approach to structured pruning of large language models.
- Addresses distribution mismatch, sign information loss, and outlier influence.
- Maintains accuracy comparable to unstructured pruning methods.
- Achieves practical inference speedup on multiple LLMs.
- Tested on Llama-3-8B, Vicuna-v1.5-13B, and LLaVA-v1.5-13B.
Paper Resources
📖 Reader Mode
~2 min readAbstract:This paper proposes an improved structured pruning method for large language models (LLMs) that addresses key challenges in adapting Adaptive Feature Retention (AFR), an unstructured pruning technique, to structured pruning. When applying AFR to structured pruning, three major problems arise: distribution mismatch between heterogeneous pruning scores, loss of sign information indicating optimization direction consistency, and influence of outliers. To address these issues, we propose a unified approach combining power transformation for nonlinear distribution alignment, sign-preserving score aggregation, and percentile-based outlier removal. Experiments on Llama-3-8B, Vicuna-v1.5-13B, and LLaVA-v1.5-13B demonstrate that our method maintains accuracy comparable to unstructured pruning while achieving practical inference speedup through structured pruning.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.08027 [cs.CL] |
| (or arXiv:2607.08027v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.08027 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yasunori Ishii Mr [view email]
[v1]
Thu, 9 Jul 2026 01:05:42 UTC (1,064 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Quantifying Prior Dominance in Systems
The study introduces the Normalized Context Utilization (NCU) metric to evaluate Retrieval-Augmented Generation (RAG) systems, revealing that Small Language Models (SLMs) outperform larger models in factual extraction. The findings indicate that traditional scaling laws yield diminishing returns, with a commercial API frequently failing against adversarial evidence due to systemic confidence collapse.