DecodeShare: Tracing the Shared Subspace of LLM Decode-Time Decisions
Quick Answer
This paper shows that The DecodeShare protocol identifies a low-dimensional shared subspace in LLMs' decode-time hidden states, significantly impacting decision performance.
Quick Take
Disturbing this subspace degrades performance more than altering prefill-derived or random subspaces. This discovery enhances activation steering reliability for downstream applications.
Key Points
- DecodeShare identifies a shared subspace in during decode time.
- Disturbing this subspace degrades performance significantly more than other subspaces.
- The shared subspace aids in reliable activation steering for LLMs.
- Evaluating steering vectors at decode time yields better results than prefill proxies.
- Code for DecodeShare is publicly available for further research.
DeepSignal Analysis
What happened
The DecodeShare protocol identifies a low-dimensional shared subspace in the hidden states of large language models (LLMs) during decoding. Disturbing this subspace significantly reduces decision performance compared to altering prefill-derived or random subspaces. This finding suggests that the shared subspace plays a crucial role in the reliability of activation steering for various applications.
Key evidence
- The DecodeShare protocol identifies a low-dimensional shared subspace in LLMs' decode-time hidden states.
- Experiments show that disturbing the shared subspace degrades decision performance more than altering prefill-derived or random subspaces.
- The shared subspace can serve as a high-leverage causal channel at decode time, enhancing activation steering reliability.
Why it matters
Understanding the shared subspace in LLMs can improve the performance of various AI applications by providing insights into how these models make decisions during decoding. This knowledge can lead to better activation steering methods, which are essential for optimizing model outputs in real-world tasks. The implications of this research could influence the design of future LLMs and their applications.
Paper Resources
Source Excerpt
(LLMs) handle many tasks with one set of parameters, but under KV-cached inference it is unclear what task-general structure, if any, is used at decode time rather than during prefill. We propose DecodeShare, a protocol that identifies a low-dimensional subspace consistently shared across tasks in decode-time hidden states, and then tests its causal role by removing that subspace only during decoding. In our experiments, disturbing the discovered shared subspace degrades de
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →AINTMA: Agentic AI Architecture for Autonomous Test Management with Generative Intelligence, Secure Cloud Communication and Adaptive Quality Analytics
AINTMA, an autonomous test management architecture utilizing six specialized AI agents, achieves 88.4% test prioritization accuracy and reduces defect escape rates from 8.3% to 2.1%. The system demonstrates a 340% ROI within nine months, showcasing the potential of agentic AI in enhancing software quality management in cloud environments.