Mix-Quant: Quantized Prefilling, Precise Decoding for Agentic LLMs
Quick Answer
Mix-Quant introduces a phase-aware quantization framework for agentic LLMs, achieving up to 3x speedup in the prefilling stage while maintaining BF16 precision during decoding.
Quick Take
Mix-Quant introduces a phase-aware quantization framework for agentic LLMs, achieving up to 3x speedup in the prefilling stage while maintaining BF16 precision during decoding. This approach alleviates the computational bottleneck in long-context, multi-turn inference, demonstrating significant efficiency improvements without substantial accuracy loss.
Key Points
- Mix-Quant leverages FP4 quantization for efficient agentic inference.
- Significant performance degradation occurs when quantizing the entire inference process.
- Prefilling stage shows quantization redundancy, allowing minimal accuracy loss.
- High-throughput NVFP4 quantization is applied to prefilling, preserving decoding quality.
- Extensive experiments confirm Mix-Quant maintains task performance with efficiency gains.
Paper Resources
📖 Reader Mode
~2 min readAbstract:LLM agents have recently emerged as a powerful paradigm for solving complex tasks through planning, tool use, memory retrieval, and multi-step interaction. However, these agentic workflows often introduce substantial input-side overhead, making the compute-intensive prefilling stage a key bottleneck in long-context, multi-turn inference. In this work, we propose Mix-Quant, a simple and effective phase-aware quantization framework for fast agentic inference. We first investigate FP4 quantization in agentic LLM workflows and observe that quantizing the entire inference process can incur significant performance degradation. In contrast, the prefilling stage exhibits substantial quantization redundancy and can therefore be quantized with minimal accuracy loss, despite being the dominant source of computation. Based on this insight, we apply high-throughput NVFP4 quantization to the prefilling phase while preserving BF16 precision for decoding. By decoupling prefilling acceleration from decoding quality, Mix-Quant combines phase-aware algorithmic quantization with hardware-efficient NVFP4 execution to alleviate the inference bottleneck in LLM agents. Extensive experiments across long-context and agentic benchmarks demonstrate that Mix-Quant largely preserves task performance while delivering significant efficiency improvements, achieving up to a 3x speedup during prefilling.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2605.20315 [cs.CL] |
| (or arXiv:2605.20315v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.20315 arXiv-issued DOI via DataCite |
Submission history
From: Haiquan Lu [view email]
[v1]
Tue, 19 May 2026 17:50:17 UTC (662 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Quantifying Prior Dominance in Systems
The study introduces the Normalized Context Utilization (NCU) metric to evaluate Retrieval-Augmented Generation (RAG) systems, revealing that Small Language Models (SLMs) outperform larger models in factual extraction. The findings indicate that traditional scaling laws yield diminishing returns, with a commercial API frequently failing against adversarial evidence due to systemic confidence collapse.