Building the foundation for running extra-large language models
Quick Answer
Cloudflare AI's Workers AI has optimized the Kimi K2.5 model, achieving 3x faster processing through prefill decode disaggregation and efficient prompt caching, improving p90 time per token from ~100 ms to 20-30 ms.
Quick Take
This architecture enhances throughput and performance for agentic applications.
Key Points
- Kimi K2.5 model processing improved by 3x with new architecture.
- Prefill decode disaggregation allows independent tuning of GPU resources.
- p90 time per token reduced from ~100 ms to 20-30 ms.
- Input token cache hit ratios increased from 60% to 80% during peak times.
- Prompt caching incentivized through discounted cached tokens for users.
Source Excerpt
We built a custom technology stack to run fast on Cloudflare’s infrastructure. This post explores the engineering trade-offs and technical optimizations required to make high-performance AI inference accessible.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from Cloudflare AI
See more →
Temporary Cloudflare Accounts for AI agents
Cloudflare introduces Temporary Accounts on Cloudflare Workers, enabling AI agents to deploy live Workers instantly using 'wrangler deploy --temporary'. This feature removes barriers for AI agents, allowing them to operate more efficiently in real-time environments.
