ThunderAgent: 2x Faster Agentic Inference for Synthetic Data Generation at Scale
Quick Answer
ThunderAgent enhances synthetic data generation by achieving 2.5× higher single-node throughput and 2.4× speedup across 8 nodes, effectively mitigating KV cache thrashing in agentic inference workflows.
Quick Take
This system is crucial for training like Claude Code and Codex that require high concurrency in multi-turn tasks.
Key Points
- Achieves over 2× single-node throughput with 10× lower P50 latency.
- Delivers 2.4× speedup on 8 nodes with near-linear scaling from 16 to 64 GPUs.
- Mitigates KV cache thrashing through program-level admission control.
- Compatible with existing offloading and speculative decoding systems.
- Accepted as a Spotlight paper at ICML 2026.
DeepSignal Analysis
What happened
ThunderAgent is a new system that improves synthetic data generation by enhancing throughput and reducing latency in agentic inference workflows. It achieves 2.5× higher single-node throughput and 2.4× speedup on an 8-node cluster, addressing issues like KV cache thrashing. This system is particularly relevant for training large language models (LLMs) that require high concurrency.
Key evidence
- ThunderAgent achieves 2.5× higher single-node throughput and 2.4× speedup on an 8-node cluster, indicating significant performance improvements.
- The system mitigates KV cache thrashing, which is a critical issue in existing inference engines when handling high concurrency in agentic workflows.
- ThunderAgent has been integrated into synthetic data generation pipelines, such as those behind datasets like CoderForge, demonstrating its practical application.
Why it matters
The advancements made by ThunderAgent are crucial for the efficient training of LLMs, which increasingly operate as agents in complex tasks. By resolving inefficiencies in memory management and request scheduling, it allows for more effective use of computational resources. This can lead to faster development cycles and improved performance in applications that rely on multi-turn interactions.
Source Excerpt
ThunderAgent is a program-aware scheduler for agentic inference. By treating each agent workflow as a schedulable program, it eliminates KV cache thrashing to deliver more than 2x single-node throughput and near-linear multi-node scaling.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from Together AI
See more →
Open, convenient and predictable: Introducing Provisioned Throughput
Together AI introduces Provisioned Throughput, offering guaranteed inference capacity for MiniMax M3 and GLM-5.2 at $0.05 per PTU per minute, achieving costs up to 90% lower than Claude Opus 4.8. This new model provides predictable pricing and a 99% uptime SLA, catering to companies transitioning to open weight models for production workloads.
