CoSA: Accelerating Long-Context Inference via Proxy-Kernel Co-Designed Sparse Attention
Quick Answer
CoSA introduces a two-stage training-free Sparse Attention mechanism that combines Kernel-Aware Proxy (KAP) and Ordered-Skipping Kernel (OSK), achieving a 4.93× speedup in attention and 2.53× reduction in Time-to-First-Token for long-context inference.
Quick Take
This method enhances accuracy under tight budgets, outperforming existing proxy-based approaches across mainstream backbones and benchmarks.
Key Points
- CoSA achieves higher accuracy at lower budgets compared to existing methods.
- The model provides a 4.93× speedup in attention computation.
- It reduces Time-to-First-Token by 2.53× with a context length of 128K.
- KAP selects blocks and produces an ordered mask for efficient KV page access.
- OSK skips more blocks using online-softmax statistics under budget constraints.
DeepSignal Analysis
What happened
CoSA introduces a new Sparse Attention mechanism that integrates Kernel-Aware Proxy (KAP) and Ordered-Skipping Kernel (OSK). This method reportedly achieves significant improvements in speed and efficiency for long-context inference tasks.
Key evidence
- CoSA achieves a 4.93× speedup in attention computation, which could enhance processing efficiency in large language models.
- The method reduces Time-to-First-Token by 2.53× under a context length of 128K, indicating faster response times for applications.
- CoSA demonstrates higher accuracy at lower budgets compared to existing proxy-based approaches across various mainstream LLM backbones and benchmarks.
Why it matters
The advancements presented by CoSA could address the growing demand for efficient long-context processing in large language models, which is critical for applications requiring real-time responses. By improving speed and accuracy, this method may facilitate broader adoption and deployment of AI technologies in resource-constrained environments.
What to watch
Paper Resources
Source Excerpt
The quadratic cost of self-attention makes long-context inference prohibitively expensive, and proxy-based block-sparse attention has become a practical remedy. Existing methods typically rely on a proxy to predict a binary sparse mask and a kernel to consume this mask and perform sparse attention computation. Such an approach is effective under moderate budgets. However, as the budget tightens, the estimated proxy inevitably drops some salient blocks, while the kernel can only apply the sparse
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.