Refusal Beyond a Single Direction: A Preliminary Comparison of Diff-in-Means and INLP
Quick Answer
This paper shows that Arditi et al.
Quick Take
Arditi et al. (2024) compare difference-in-means (DiM) and Iterative Nullspace Projection (INLP) methods for refusal in chat models, finding INLP's counterfactual flipping is competitive with DiM's directional ablation. While nullspace projection is less effective, restricting INLP to leading directions maintains suppression effects, suggesting distinct activation space behaviors.
Key Points
- INLP's counterfactual flipping matches DiM's effectiveness in refusal suppression.
- Nullspace projection consistently shows weaker performance compared to DiM methods.
- Restricting INLP to leading directions preserves suppression effects at near-baseline perplexity.
- Distinct activation behaviors suggest models encode absence of concepts differently.
- Further investigation into these distinctions is warranted for future research.
Paper Resources
Article Content
From source RSS / original summaryarXiv:2606. 13720v1 Announce Type: new Abstract: Arditi et al. (2024) has shown that refusal in safety fine-tuned chat models is mediated by a single linear direction in the residual stream, recoverable by a difference-in-means (DiM) of harmful and harmless activations.
We compare DiM-based interventions (activation addition and directional ablation) with two interventions derived from Iterative Nullspace Projection (INLP) -- nullspace projection and counterfactual flipping -- on five open-weight chat models, asking whether INLP can match DiM at steering refusal and whether its richer parameterisation yields more tweakable interventions.
INLP counterfactual flipping is competitive with DiM directional ablation on refusal suppression, while nullspace projection is consistently weaker. Restricting INLP to the leading directions of the extracted subspace preserves most of the suppression effect at near-baseline perplexity, giving a tunable capability.
Geometrically, the two INLP interventions land in qualitatively different regions of activation space: nullspace projection collapses transformed activations \emph{between} the harmful and harmless clusters, while counterfactual flipping moves them into the opposite cluster, suggesting that the model encodes the absence of a concept differently from its opposite -- an intriguing distinction that warrants further investigation in future work.
Reader Mode unavailable (could not extract clean content).
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →Arbor: Tree Search as a Cognition Layer for Autonomous Agents
Arbor introduces a multi-agent framework utilizing structured tree search for optimizing LLM inference, achieving up to 193% throughput-latency improvement compared to vendor-optimized systems. It employs an Orchestrator and Critic agent for stability and coordination, demonstrating hardware-agnostic performance with minimal variance.