The Calibration Floor: Format Repair Can Masquerade as Self-Correction at Small-to-Mid Scale
Quick Answer
The study reveals that format repair often mimics self-correction in language models like Qwen3.5 and Gemma-4-12B, with format effects dominating content effects in 12 out of 29 tested cells.
Quick Take
Causal testing indicates that grammar-constrained decoding can close 71% of the accuracy gap, highlighting the need for better calibration in model evaluations.
Key Points
- Format effects exceed content effects in 12 out of 29 primary cells tested.
- Causal testing shows grammar-constrained decoding closes 71% of accuracy gap.
- Floor-scale models have higher odds of content-level change than capable-scale models.
- Total effects can reach +0.275, yet content margin is zero in larger models.
- The study provides instruments, code, and results for further research.
DeepSignal Analysis
What happened
The study investigates the phenomenon of format repair in language models, specifically Qwen3.5 and Gemma-4-12B. It finds that format effects often overshadow content effects, particularly in 12 out of 29 tested scenarios. Causal testing reveals that grammar-constrained decoding can significantly reduce the accuracy gap attributed to format issues.
Key evidence
- In 12 out of 29 tested cells, format effects were found to exceed content effects, with a Wilcoxon p-value of 1.7e-3.
- Causal testing showed that grammar-constrained decoding closed a median of 71% of the accuracy gap between naive total effects and content-margin estimates across 14 cells.
- The study indicates that floor-scale models (0.8B/2B) have a higher likelihood of experiencing content-level changes compared to capable-scale models, with a p-value of less than 1e-7.
Why it matters
Understanding the distinction between format repair and genuine self-correction is crucial for evaluating language models. The findings suggest that many perceived improvements in model accuracy may not stem from enhanced reasoning capabilities but rather from adjustments in output format. This has implications for how models are assessed and calibrated, potentially leading to misinterpretations of their performance.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Accuracy changes after language-model self-revision are usually interpreted as changes in reasoning. We show this can fail at the answer-extraction boundary, and test the failure causally rather than only observationally. Across Qwen3.5 (0.8B-9B), Gemma-4-12B, and two frontier models via API (Tencent Hy3, Nvidia Nemotron-3-Ultra-550B) in 29 primary cells plus a frontier arm, we decompose the always-revise accuracy shift into a content margin (both answers parseable) and format-recovery/loss margins (parseability changes). On 12 cells with meaningful unparseable-answer rates, format effects exceed content effects (Wilcoxon p=1.7e-3). To test this causally, we force already-generated reasoning through grammar-constrained decoding so every answer is parseable by construction: across 14 cells this closes a median 71% of the gap between the naive total effect and the content-margin estimate, with two cells converging exactly and a residual on the two largest-effect cells reported rather than dismissed. A clustered model confirms floor-scale (0.8B/2B) models have far higher odds of content-level change and harm than capable-scale models (p<1e-7). Replicating a cited confidence-gating protocol verbatim on Qwen3.5 does not reproduce its reported gain and shows the same near-zero content margin. A frontier check on much larger models shows format-dominance intensifying with scale: content margin is exactly zero in all 5 cells despite total effects up to +0.275, though this arm is lower-powered. The calibration-floor criterion on the content margin reveals a squeeze: floor-scale cells have headroom but insufficient signal, capable-scale cells have signal but little headroom; only one cell is marginally viable, with negligible sealed-holdout gain. Content is a minority share of what the field has measured as self-correction. We release the instrument, code, and derived results.
| Comments: | 36 pages, 5 figures |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2608.04355 [cs.CL] |
| (or arXiv:2608.04355v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2608.04355 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Mingguang Chen [view email]
[v1]
Wed, 5 Aug 2026 01:55:33 UTC (517 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.