LISA: Linear-Indexed Sparse Attention for Efficient Long-Context Reasoning
Quick Answer
This paper shows that LISA (Linear-Indexed Sparse Attention) enhances long-context reasoning by reducing inference complexity from O(n^2) to O(nM) using a Linear Attention module and a Lightning Indexer.
Quick Take
Experiments show a 50% speedup in inference with 16K-token contexts and a 5.6% performance improvement on benchmarks like AIME and MATH-500.
Key Points
- LISA integrates Linear Attention and Lightning Indexer for efficient long-context processing.
- Reduces inference complexity significantly, enabling better scalability in production settings.
- Achieves 50% faster inference speeds on 16K-token contexts.
- Improves reasoning performance by 5.6% on benchmarks like AIME and MATH-500.
- Utilizes a two-stage training pipeline for optimal model integration.
Paper Resources
Source Excerpt
Recent advances in long chain-of-thought reasoning models such as DeepSeek-R1 have led to increasingly longer inference context lengths under the paradigm. However, the O(n^2) computational complexity of standard self-attention causes inference costs to grow sharply with long sequences, limiting the deployment of long-CoT reasoning in production settings. To address this, we propose LISA (Linear-Indexed Sparse Attention), a plug-and-play attention replacement module that requir
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →RAIL Guard: Closing the Evaluation-to-Remediation Gap in Responsible AI for Agents
RAIL Guard introduces a closed-loop AI pipeline for large language models (LLMs) that evaluates outputs across eight dimensions and iteratively remediates failures, achieving 96.9% convergence compared to 49.1% for traditional block-and-retry methods. The system reduces unsafe agent executions by 33% without impacting task completion and is available as open-source SDKs.