DiscoLoop: Looping Discrete Embeddings and Continuous Hidden States for Multi-hop Reasoning
Quick Answer
DiscoLoop introduces a novel looping architecture that combines discrete embeddings and continuous hidden states, achieving near-perfect accuracy in multi-hop reasoning tasks with fewer training steps.
Quick Take
This model outperforms looped-transformer baselines in real-world pretraining, demonstrating lower training loss and enhanced benchmark performance.
Key Points
- DiscoLoop addresses the depth-local storage problem in standard Transformers.
- The model achieves near-perfect accuracy in symbolic and synthetic-language reasoning tasks.
- It shows significant performance improvements with fewer training steps compared to existing models.
- DiscoLoop's mixed-channel design enhances practical language modeling capabilities.
- An easy training-free realignment intervention significantly reduces the generalization gap.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Large language models achieve strong performance on many reasoning tasks when allowed to externalize intermediate steps as Chain-of-Thought (CoT). However, many questions require the model to internalize the multi-step reasoning within a single forward pass before generating the answer. We study this challenge through two-hop reasoning, a representative task where the model must compose multiple pieces of parametric knowledge within a single forward pass. Standard non-recurrent Transformers suffer from a depth-local storage problem: facts learned in earlier layers are unavailable where second-hop retrieval happens. We found that Looped Transformers mitigate this issue by reusing the same memory, but still generalize imperfectly. We show that the remaining bottleneck is representational. In the two-hop reasoning task, the first loop often makes the correct bridge entity nearly perfectly decodable, yet the corresponding hidden state remains poorly aligned with the bridge token embedding. Surprisingly, an easy training-free realignment intervention nearly closes the generalization gap. Building upon this insight, we propose DiscoLoop, a looping architecture whose recurrence carries both a discrete embedding channel and a continuous hidden-state channel. DiscoLoop achieves near-perfect accuracy with substantially fewer training steps across symbolic and synthetic-language multi-hop reasoning tasks. When applied to real-world pretraining, DiscoLoop attains lower training loss and stronger benchmark performance than looped-transformer baselines, suggesting that the mixed-channel design transfers to practical language modeling.
| Comments: | 16 pages, 7 figures |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2607.00341 [cs.CL] |
| (or arXiv:2607.00341v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.00341 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Hengyu Fu [view email]
[v1]
Wed, 1 Jul 2026 02:32:02 UTC (1,318 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.