Hidden Decoding at Scale: Latent Computation Scaling for Large Language Models
Quick Answer
The paper introduces Hidden Decoding, a novel sequence-length scaling method for Large Language Models (LLMs) like WeLM-HD4-80B and WeLM-HD4-617B, enhancing performance without altering the Transformer backbone.
Quick Take
The paper introduces Hidden Decoding, a novel sequence-length scaling method for Large Language Models (LLMs) like WeLM-HD4-80B and WeLM-HD4-617B, enhancing performance without altering the Transformer backbone. By utilizing Stream-Factorized Attention, it reduces attention costs from quadratic to linear, demonstrating effective scaling at the 100B+ MoE level.
Key Points
- Hidden Decoding expands each token into n streams with independent embedding tables.
- Stream-Factorized Attention reduces attention costs from quadratic to linear in n.
- WeLM-HD4-80B and WeLM-HD4-617B show improved performance over non-HD baselines.
- The method allows continued pretraining without additional Transformer layers.
- Scaling results indicate gains increase with larger expansion factors.
Paper Resources
📖 Reader Mode
~5 min read
Computer Science > Computation and Language
arXiv:2607.08186 (cs)
Authors:Aiwei Liu, Cheng Shi, Chuhan Wu, Ci Lei, Di Lu, Donald He, Fan Zhang, Fanhao Kong, Feifei Zhang, Guan Wang, Haicheng Wang, Haoyu Liu, Houjin Yu, Jiachen Ding, Jiayi Feng, Jie Zhou, Jijun Chi, Jindi Shi, Jing Lei, Junjie Zhang, Laiyi Li, Le Tian, Linhao Zhang, Miao Fan, Sijun Zhang, Wei Jia, Weiwei Shi, Wenhan Li, Wentao Zhao, Wenteng Liang, Xiao Zhou, Xiaojin Zhou, Xihuai Wang, Xinyu Gao, Xuanliang Wang, Xuyang Ao, Yang Yu, Yangxiu You, Yinuo Zhao, Yufei Kuang, Yufei Wang, Yuan Liu, Yuan Liu, Yuwen Chen, Zhencong Tian, Zhongyin Zhao, Zilin Yu, Zitao Wang
Abstract:Scaling Large Language Models (LLMs) has been driven mainly by enlarging the Transformer backbone, but for an already-strong model this requires another round of costly pretraining. We study whether an existing backbone can keep improving by allocating more computation to each token while leaving the Transformer backbone fixed. Depth-recurrent (looped) Transformers pursue this goal but are hard to scale, because looped computation does not fit naturally with the pipeline parallelism used to train the largest models. We add computation along the sequence-length dimension, where the extra computation is simply a longer input and stays compatible with standard large-model training. We propose Hidden Decoding, a sequence-length scaling method applied during continued pretraining (CPT). It expands each token into n streams with independent embedding tables and keeps the intermediate streams' key-value cache as context, so each token performs more internal computation without adding or widening Transformer layers. To keep this affordable at scale, we introduce Stream-Factorized Attention, in which most layers attend only within each stream and only a few layers mix across streams, reducing the attention cost from quadratic to roughly linear in n. Experiments support two scaling results. At frontier scale, we train WeLM-HD4-80B and WeLM-HD4-617B at n=4 and improve their matched non-HD baselines, making Hidden Decoding the first demonstrated sequence-length scaling method at the 100B+ MoE scale. Across expansion factors, the gains grow as n increases, showing that sequence-length expansion is a practical fixed-backbone scaling path for frontier-scale LLMs.
| Comments: | 30 pages, 9 figures |
| Subjects: | Computation and Language (cs.CL) |
| MSC classes: | 68T50 |
| ACM classes: | I.2.7 |
| Cite as: | arXiv:2607.08186 [cs.CL] |
| (or arXiv:2607.08186v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.08186 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Aiwei Liu [view email]
[v1]
Thu, 9 Jul 2026 07:37:59 UTC (2,687 KB)
Current browse context:
cs.CL
Change to browse by:
Bookmark
Bibliographic and Citation Tools
Code, Data and Media Associated with this Article
Demos
Recommenders and Search Tools
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Quantifying Prior Dominance in Systems
The study introduces the Normalized Context Utilization (NCU) metric to evaluate Retrieval-Augmented Generation (RAG) systems, revealing that Small Language Models (SLMs) outperform larger models in factual extraction. The findings indicate that traditional scaling laws yield diminishing returns, with a commercial API frequently failing against adversarial evidence due to systemic confidence collapse.