Smaller, faster, safer: running Kimi and GLM at scale
Quick Answer
Cloudflare's Workers AI optimizes GPU memory usage for Kimi K-series and GLM models by quantizing KV caches and compressing model weights, achieving up to 41% higher throughput and 40% reduced memory usage without sacrificing accuracy.
Quick Take
This allows for more concurrent requests, enhancing service efficiency and lowering costs for customers.
Key Points
- Quantizing KV cache from BF16 to FP8 doubles context capacity from 686,000 to 1.37 million tokens.
- Model weight compression from 8-bit to 4-bit reduces GLM 5.2 size by 40%, freeing GPU memory.
- FP8 KV cache supports 64 concurrent requests, achieving 2,192 tokens/sec, 41% higher than BF16.
- Model accuracy remains consistent across benchmarks, with negligible differences between FP8 and INT4.
- SGLang framework ensures optimal performance for inference serving, benefiting the open-source community.
DeepSignal Analysis
What happened
Cloudflare's Workers AI has implemented optimizations for the Kimi K-series and GLM models, enhancing GPU memory efficiency. Techniques such as quantizing KV caches and compressing model weights have led to a 41% increase in throughput and a 40% reduction in memory usage, while maintaining accuracy. These improvements allow for more concurrent requests, ultimately benefiting customers by lowering operational costs.
Key evidence
- The KV cache size was reduced from 16-bit to 8-bit floating point, doubling the context capacity from 686,000 tokens to 1.37 million.
- Model weights for GLM 5.2 were compressed from 8-bit floating point to 4-bit integers, reducing the checkpoint size from 705 GB to 421 GB.
- KV cache integrity checking was implemented, resulting in less than 1% change in throughput and tail latency, ensuring request accuracy.
Why it matters
These advancements in memory optimization are crucial for efficiently serving large AI models, which are often limited by GPU memory constraints. By enabling higher throughput and reduced memory usage, Cloudflare can support more users simultaneously, enhancing service delivery and reducing costs. This is particularly significant as demand for AI services continues to grow, making efficient resource management essential.
Source Excerpt
Serving frontier models like Kimi and GLM means fighting for GPU memory. Here's how we quantize KV caches, compress model weights, and add integrity checks to serve them faster, cheaper, and safely.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from Cloudflare AI
See more →
Temporary Cloudflare Accounts for AI agents
Cloudflare introduces Temporary Accounts on Cloudflare Workers, enabling AI agents to deploy live Workers instantly using 'wrangler deploy --temporary'. This feature removes barriers for AI agents, allowing them to operate more efficiently in real-time environments.
