Stuck on "A": Diagnosing and Repairing Interface Injury in Attention-to-KDA Linearization of a 0.6B Language Model
Quick Answer
The study converts 21 of 28 full-attention layers of the Qwen3-0.6B-Base model to KDA linear-attention layers, revealing a significant interface injury where the model predicts option labels rather than content, achieving only 25-29% accuracy.
Quick Take
A targeted KL distillation stage improved performance by 12.48 points on C-Eval, demonstrating the challenges of model conversion on a consumer-grade GPU.
Key Points
- 21 out of 28 layers of Qwen3-0.6B-Base converted to KDA linear-attention.
- Model accuracy remains near random chance at 25-29% on C-Eval.
- Interface injury causes model to predict labels instead of content.
- 1,000-step KL stage improves C-Eval score by 12.48 points.
- Engineering lessons include a silent failure mode in bf16 optimizer updates.
DeepSignal Analysis
What happened
The study converted 21 out of 28 layers of the Qwen3-0.6B-Base model to KDA linear-attention layers. This conversion resulted in a significant interface injury, where the model exhibited low accuracy in predicting content, achieving only 25-29% accuracy on multiple-choice tasks. A subsequent KL distillation stage improved performance by 12.48 points on the C-Eval benchmark.
Key evidence
- The Qwen3-0.6B-Base model had 21 of its 28 full-attention layers converted to KDA linear-attention layers.
- Post-conversion, the model's multiple-choice accuracy was between 25-29%, significantly lower than the teacher model's 50.6%.
- A targeted KL distillation stage improved the model's performance by 12.48 points on the C-Eval benchmark.
Why it matters
This research highlights the challenges faced when converting attention mechanisms in language models, particularly on consumer-grade hardware. The significant drop in accuracy indicates that the interface injury affects the model's ability to understand content, which is critical for practical applications. The findings also emphasize the importance of targeted distillation techniques in mitigating these issues, providing insights for future model optimization.
Paper Resources
📖 Reader Mode
~2 min readAbstract:We convert 21 of 28 full-attention layers of Qwen3-0.6B-Base into KDA (Kimi Delta Attention) linear-attention layers on a single consumer-grade GPU budget, and ask a simple question: what exactly does the conversion break? After surgery, hidden-state alignment and end-to-end KL distillation drive the student close to its teacher in perplexity, yet multiple-choice accuracy stays near random chance (25-29% vs. the teacher's 50.6% on C-Eval). Using a four-permutation diagnostic that rotates answer options while holding content fixed, we show the model sticks to option labels (predicting "A" 81% of the time; 106/161 questions keep the same label under all four rotations) rather than following answer content -- an interface injury that standard distillation metrics cannot see. A 1,000-step format-targeted completion-only KL stage repairs the interface (+12.48 points on C-Eval, label-stickiness roughly halved), after which persona SFT and one round of on-policy DPO preserve benchmark scores within noise. We release code, weights, recipes, and the full audit trail, and distill the engineering lessons -- including an FP32-master failure mode in which bf16 optimizer updates are silently swallowed -- that made convergence possible at this budget.
| Comments: | Code and models: this https URL ; this https URL . A version of this preprint is archived on Zenodo (DOI: https://doi.org/10.5281/zenodo.21722356) |
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2608.02689 [cs.CL] |
| (or arXiv:2608.02689v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2608.02689 arXiv-issued DOI via DataCite |
Submission history
From: Ronglong Bao [view email]
[v1]
Mon, 3 Aug 2026 08:54:18 UTC (26 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.
