Linear Probes Detect Task Format, Not Reasoning Mode in Language Model Hidden States
Quick Answer
This paper shows that Linear probing of the Qwen3-14B model reveals that high accuracy in distinguishing reasoning types is influenced by task format rather than underlying computational structures.
Quick Take
Probes achieved 100% accuracy on benchmarks like LogiQA 2.0, but residualizing factors like source identity reduced accuracy to chance levels, indicating shared reasoning across tasks.
Key Points
- Linear probes on Qwen3-14B achieved 100% accuracy on LogiQA 2.0, ARC-Challenge, and αNLI.
- Accuracy drops to chance levels when controlling for source identity and response length.
- Trace-anchor similarity indicates 42.5% agreement in reasoning across tasks.
- Causal steering shows no link between geometry and reasoning mode with p=0.286.
- Findings suggest the need for routine format deconfounding in mechanistic interpretability.
Paper Resources
Article Excerpt
From source RSS / original summaryarXiv:2606. 02907v1 Announce Type: new Abstract: Linear probing of (LLM) hidden states is widely used to claim that models learn distinct representations for different reasoning types. We test this by probing Qwen3-14B on three benchmarks spanning the classical trichotomy: LogiQA 2. 0 (deductive), ARC-Challenge (inductive), and $\alpha$NLI (abductive). At layer 32 of 40, linear probes achieve 100\% cross-validated accuracy with well-separated geometry (intrinsic dimensionalities: 20. 6, 28.
5, 33. 6; convex hull contamination $\leq$1. 5\%). However, this separation is entirely driven by format confounds. Residualizing source identity, option count, and response length reduces accuracy to chance. Trace-anchor similarity indicates largely shared reasoning across tasks (42. 5\% agreement vs. \ 33. 3\% chance), and causal steering with random controls ($n=20$) shows no functional link between geometry and reasoning mode ($p=0. 286$).
Thus, high probe accuracy reflects task format rather than computational structure, motivating routine format deconfounding in mechanistic interpretability.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.