Why Limit the Residual Stream to Layers and Not Tokens? Persistent Memory for Continuous Latent Reasoning
Quick Answer
The AGCLR model enhances the CoCoNuT paradigm by introducing a Gated Concept Stream, addressing the concept bottleneck in LLMs.
Quick Take
The AGCLR model enhances the CoCoNuT paradigm by introducing a Gated Concept Stream, addressing the concept bottleneck in LLMs. This innovation allows for persistent memory across reasoning passes, leading to improved performance on benchmarks like GSM8K and HotpotQA, with AGCLR outperforming vanilla CoCoNuT by resolving critical fact loss during reasoning. Code is available for further exploration.
Key Points
- AGCLR introduces a Gated Concept Stream to retain critical facts during reasoning.
- Performance improved on GSM8K and HotpotQA benchmarks compared to vanilla CoCoNuT.
- Concept bottleneck identified as a limitation in existing LLM reasoning methods.
- Three learned gates (write, read, forget) control memory management in AGCLR.
- Code for AGCLR is publicly available for further research and application.
Article Content
From source RSS / original summaryarXiv:2606. 07720v1 Announce Type: new Abstract: Large language models (LLMs) have demonstrated remarkable reasoning abilities on mathematical and multi-hop planning tasks. The CoCoNuT (Chain of Continuous Thought) paradigm~\cite{hao2024coconut} extends this by enabling models to reason in latent space, exploring multiple reasoning paths simultaneously rather than committing to a single chain early on. However, we identify a limitation we term the \textbf{concept bottleneck}.
At each reasoning pass, intermediate hidden states are overwritten, causing the model to lose critical facts computed in earlier steps as reasoning depth increases. We observe this empirically. On HotpotQA, vanilla CoCoNuT (10. 4\% EM) fails to improve over the CoT baseline (11. 0\% EM), and performance degrades with curriculum depth on GSM8K. To address this, we propose \textbf{AGCLR} (Adaptive Gated Continuous Latent Reasoning), which augments CoCoNuT with a \textit{Gated Concept Stream}.
A persistent residual memory maintained across all reasoning passes, controlled by three learned gates: a \textit{write} gate that commits intermediate facts to memory, a \textit{read} gate that retrieves relevant prior states, and a \textit{forget} gate that prunes irrelevant context. Evaluated on GSM8K, HotpotQA, and ProsQA using GPT-2 as our base model, AGCLR achieves consistent improvements across all types of datasets.
With the performance gap compounding as curriculum depth increases, directly resolving the concept bottleneck. Code available at https://anonymous. 4open. science/r/JJJJ/README. md
Reader Mode unavailable (could not extract clean content).
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →The Sim-to-Real Gap of Foundation Model Agents: A Unified MDP Perspective
This paper addresses the sim-to-real gap for foundation model agents by framing it within a Markov Decision Process (MDP) structure. It advocates for established solutions like domain randomization to enhance agent robustness, aiming to create standardized benchmarks for reliable real-world applications.