Silent Failures in Quantized LLM Reasoning: A Taxonomy-Based Analysis of Hollow Convergence and Failure Mode Shifts
Quick Answer
This paper shows that Post-training quantization alters reasoning in LLMs, revealing Hollow Convergence and failure shifts, particularly in smaller models.
Quick Take
Post-training quantization alters reasoning in LLMs, revealing Hollow Convergence and failure shifts, particularly in smaller models. NF4 precision causes significant drops in reasoning quality, with Shortcut Collapse rising to 78% in LLaMA 3.2-3B, while accuracy metrics fail to capture these issues.
Key Points
- Hollow Convergence occurs with NF4, especially in models below 12B parameters.
- GSM8K shows immunity to reasoning shifts, unlike LogiQA and ARC-Challenge.
- Shortcut Collapse increases from 44% to 78% in LLaMA 3.2-3B under NF4.
- Confidence Snowballing drops from 15.8% to near zero with NF4 quantization.
- Hollow Convergence detection is unreliable using surface-level text features.
Paper Resources
📖 Reader Mode
~2 min readAbstract:We show that post-training quantization can silently alter how large language models reason even when task accuracy is preserved. Using a six-category failure taxonomy validated by two independent human annotators (Cohen's $\kappa$ = 0.906), we classify 30,000 chain-of-thought outputs from five instruction-tuned LLMs (3B--14B parameters) across three quantization precisions (FP32, FP16, NF4) and four reasoning benchmarks. We find that while accuracy is robust across precisions (maximum 3.1 pp drop), Hollow Convergence (correct answers reached through incomplete or unverifiable reasoning) shows a significant size-dependent shift under NF4, dropping sharply for the two smallest models tested but remaining invariant for models at 12B parameters and above. This effect is also benchmark-specific: GSM8K is categorically immune while LogiQA and ARC-Challenge show the largest shifts. Furthermore, under NF4, Shortcut Collapse rises from 44% to 78% of wrong-answer failures in LLaMA 3.2-3B while Confidence Snowballing collapses from 15.8% to near zero, a qualitative shift invisible to accuracy metrics. Finally, we show Hollow Convergence cannot be reliably detected from surface-level text features (best F1 = 0.53), establishing it as a deployment-relevant failure mode that standard evaluation pipelines cannot catch.
| Comments: | 7 pages, 3 figures, 6 tables |
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| ACM classes: | I.2.7; I.2.6 |
| Cite as: | arXiv:2607.09999 [cs.CL] |
| (or arXiv:2607.09999v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.09999 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Renuka Oladri [view email]
[v1]
Fri, 10 Jul 2026 21:55:05 UTC (467 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Letting the Data Speak: Extracting Keywords from Crowdsourced Collections with AI
The study evaluates three NLP approaches—Named Entity Recognition, Keyword Extraction, and Topic Modelling—using the Their Finest Hour Online Archive to automate keyword extraction from crowdsourced WWII collections. Findings suggest that while NLP methods show promise, no single approach is sufficient, and ethical considerations in automated keyword extraction are crucial for responsible stewardship.