Minimizing the Hidden Cost of Scales: Graph-Guided Ultra-Low-Bit Quantization for Large Language Models
Quick Answer
SAGE-PTQ introduces a novel ultra-low-bit quantization framework for large language models, achieving 1.03 weight bits and 0.004 scaling bits per matrix, significantly outperforming BiLLM and PB-LLM.
Quick Take
On LLaMA-3-8B, it achieves a perplexity of 6.74, compared to BiLLM's 55.8, while using less than 50% of BiLLM's GPU memory and demonstrating 1.5x faster decoding on LLaMA-2-70B with a single NVIDIA L40 GPU.
Key Points
- SAGE-PTQ minimizes hidden scaling costs in ultra-low-bit quantization for .
- Achieves 1.03 weight bits and 0.004 scaling bits per matrix on average.
- Outperforms BiLLM with a perplexity of 6.74 on LLaMA-3-8B.
- Uses less than 50% of BiLLM's GPU memory for similar tasks.
- Demonstrates 1.5x faster decoding on LLaMA-2-70B with NVIDIA L40 GPU.
Paper Resources
Source Excerpt
arXiv:2606. 05429v1 Announce Type: new Abstract: Post-training quantization (PTQ) is critical for the efficient deployment of (LLMs). Recent ultra-low-bit PTQ methods rely on rigid weight-saliency assumptions or position heuristics, introducing substantial hidden scaling overhead. We propose SAGE-PTQ (Saliency-Aware Graph-guided Efficient PTQ), a novel ultra-low-bit quantization framework for LLMs that minimizes hidden scaling cost.
SAGE-PTQ separates salient and unsalient weights using distributional statistics, then models subsampled unsalient weights as a sparse graph to estimate the optimal number of groups per layer. …
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →RAIL Guard: Closing the Evaluation-to-Remediation Gap in Responsible AI for Agents
RAIL Guard introduces a closed-loop AI pipeline for large language models (LLMs) that evaluates outputs across eight dimensions and iteratively remediates failures, achieving 96.9% convergence compared to 49.1% for traditional block-and-retry methods. The system reduces unsafe agent executions by 33% without impacting task completion and is available as open-source SDKs.
