Kernel Forge: An Agent Harness for LLM-based Generation and Optimization of CUDA Kernels
Quick Answer
Kernel Forge is an open-source agentic harness that optimizes CUDA kernels for PyTorch models using MCTS, achieving up to 2.83x performance improvements on specific workloads.
Quick Take
It supports vision, diffusion, and tasks, significantly reducing the need for expert intervention in kernel optimization. The tool has been evaluated on NVIDIA DGX Spark with impressive results across multiple models.
Key Points
- Optimizes CUDA kernels for PyTorch models with only 50 iterations per kernel.
- Achieves 1.52x speedup on adaptive_avgpool2d in ResNet-50.
- Delivers 2.83x improvement on softmax in Gemma 4 E2B.
- Utilizes Monte Carlo Tree Search for exploring optimization paths.
- Includes a GUI for monitoring and debugging kernel performance.
DeepSignal Analysis
What happened
Kernel Forge is an open-source tool designed to optimize CUDA kernels for PyTorch models using Monte Carlo Tree Search (MCTS). It has demonstrated performance improvements of up to 2.83 times on specific workloads, evaluated on NVIDIA DGX Spark. The tool supports various tasks, including vision and diffusion, while reducing the need for expert intervention in kernel optimization.
Key evidence
- Kernel Forge optimizes CUDA kernels for PyTorch models, achieving up to 2.83 times performance improvements on specific workloads.
- The tool has been evaluated on NVIDIA DGX Spark with 50 optimization iterations per kernel, outperforming PyTorch eager mode in multiple cases.
- Kernel Forge supports vision, diffusion, and LLM workloads, utilizing MCTS to explore multiple optimization paths.
Why it matters
The development of Kernel Forge represents a significant advancement in automating the optimization of GPU compute kernels, which are critical for machine learning performance. By reducing the reliance on expert engineers, it could democratize access to high-performance computing for a broader range of developers. This could lead to faster iterations in model development and deployment, ultimately benefiting various applications in AI.
Paper Resources
Source Excerpt
Machine learning models are increasingly embedded in everyday software, and most of their runtime is spent in a small set of compute kernels such as matrix multiplication, convolution, and normalization. Optimizing these kernels is one of the most direct ways to reduce latency and cost, but it has traditionally required expert engineers to hand-write low-level GPU code. Agentic systems built on (LLMs) can now generate and optimize kernels with far less human effort, yet exi
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
HOBA (Hierarchical On-policy Bidding Agents) is a novel hierarchical reinforcement learning framework that enhances online advertising bidding systems by improving adaptability and reducing hyperparameter tuning costs. It utilizes a for hyperparameter inference, a SARSA agent for expert model selection, and a dynamic expert pool for bid execution, achieving a +3.6% increase in target cost during large-scale deployment and outperforming state-of-the-art baselines on AuctionNet.
