Mergeable Model-Side Aggregation States for Long-Context Language Models
Quick Answer
The paper introduces a model-side aggregation interface using Hash-based HyperLogLog (HLL) states to enhance long-context language models like Gemma 4 and Qwen, achieving up to 99.3% accuracy on aggregate-then-reason tasks.
Quick Take
This approach maintains a fixed HLL state size, allowing for efficient merging of states across segments, significantly improving performance over traditional reasoning methods by up to 63.2 points.
Key Points
- Introduces a compact aggregation interface for long-context models using HLL sketch states.
- Achieved 99.2% accuracy on 3,969 aggregate-then-reason tasks with Gemma 4.
- Fixed HLL state size of 2 KiB remains constant regardless of context length.
- Improved performance by 63.2 points over direct full-context reasoning on matched sets.
- Code for the proposed method is publicly available.
DeepSignal Analysis
What happened
The paper presents a model-side aggregation interface that utilizes Hash-based HyperLogLog (HLL) states to improve long-context language models. This method allows for efficient merging of states across segments, achieving high accuracy on various reasoning tasks.
Key evidence
- The proposed method reached 99.2% accuracy on 3,969 aggregate-then-reason tasks using Gemma 4, with a minimal gap of 0.8 percentage points compared to exact aggregation.
- In distinct-count experiments with one million records, the mean relative error was reported at 1.6%, demonstrating the effectiveness of the HLL state.
- The approach improved performance over direct full-context reasoning by 63.2 points on Qwen and 56.3 points on Gemma, indicating significant enhancements in reasoning tasks.
Why it matters
This advancement addresses a known limitation of long-context language models, which struggle with set-based aggregation as context length increases. By maintaining a fixed HLL state size, the method enhances efficiency and accuracy, making it applicable to various real-world tasks such as logs and multi-turn conversations.
What to watch
Paper Resources
Source Excerpt
A known limitation of long-context language models is their increasingly unreliable performance in non-additive, set-based aggregation as context length grows. Examples include cardinality estimation, set relationships, and grouped statistics, which widely exist in logs, program outputs, tables, and multi-turn conversations. To provide the aggregation state required by these tasks, we introduce a model-side aggregation interface that maintains compact Hash-based HyperLogLog (HLL) sketch states a
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.