Kernel Forge: An Agent Harness for LLM-based Generation and Optimization of CUDA Kernels
Quick Answer
Kernel Forge is an open-source agentic harness that optimizes CUDA kernels for PyTorch models using MCTS, achieving up to 2.83x performance improvements on specific workloads.
Quick Take
It supports vision, diffusion, and tasks, significantly reducing the need for expert intervention in kernel optimization. The tool has been evaluated on NVIDIA DGX Spark with impressive results across multiple models.
Key Points
- Optimizes CUDA kernels for PyTorch models with only 50 iterations per kernel.
- Achieves 1.52x speedup on adaptive_avgpool2d in ResNet-50.
- Delivers 2.83x improvement on softmax in Gemma 4 E2B.
- Utilizes Monte Carlo Tree Search for exploring optimization paths.
- Includes a GUI for monitoring and debugging kernel performance.
DeepSignal Analysis
What happened
Kernel Forge is an open-source tool designed to optimize CUDA kernels for PyTorch models using Monte Carlo Tree Search (MCTS). It has demonstrated performance improvements of up to 2.83 times on specific workloads, evaluated on NVIDIA DGX Spark. The tool supports various tasks, including vision and diffusion, while reducing the need for expert intervention in kernel optimization.
Key evidence
- Kernel Forge optimizes CUDA kernels for PyTorch models, achieving up to 2.83 times performance improvements on specific workloads.
- The tool has been evaluated on NVIDIA DGX Spark with 50 optimization iterations per kernel, outperforming PyTorch eager mode in multiple cases.
- Kernel Forge supports vision, diffusion, and LLM workloads, utilizing MCTS to explore multiple optimization paths.
Why it matters
The development of Kernel Forge represents a significant advancement in automating the optimization of GPU compute kernels, which are critical for machine learning performance. By reducing the reliance on expert engineers, it could democratize access to high-performance computing for a broader range of developers. This could lead to faster iterations in model development and deployment, ultimately benefiting various applications in AI.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Machine learning models are increasingly embedded in everyday software, and most of their runtime is spent in a small set of compute kernels such as matrix multiplication, convolution, and normalization. Optimizing these kernels is one of the most direct ways to reduce latency and cost, but it has traditionally required expert engineers to hand-write low-level GPU code. Agentic systems built on large language models (LLMs) can now generate and optimize kernels with far less human effort, yet existing tools are largely evaluated on randomly generated tensors and isolated kernels, emit standalone CUDA code that developers must manually reintegrate, mostly target only LLM PyTorch models, and offer limited support for inspecting and debugging results. We present Kernel Forge, an open-source, end-to-end agentic harness that accepts any unmodified PyTorch model in place. Kernel Forge supports vision, diffusion, and LLM workloads, uses Monte Carlo Tree Search (MCTS) to explore multiple optimization paths rather than a single linear refinement chain, and ships with a graphical user interface for monitoring progress, inspecting candidate kernels, and debugging failures. We evaluate Kernel Forge on four PyTorch models spanning vision, diffusion, and LLM workloads on an NVIDIA DGX Spark with GB10 GPU. With only 50 optimization iterations per kernel, it optimizes 14 kernels to outperform PyTorch eager mode, reaching $1.52\times$ on adaptive\_avgpool2d in ResNet-50, $1.70\times$ on group\_norm in Stable Diffusion 3.5 Medium, $2.83\times$ on softmax in Gemma 4 E2B, and $1.54\times$ on softmax in Qwen 3.5 35B-A3B.
| Comments: | The code is available at: this https URL |
| Subjects: | Artificial Intelligence (cs.AI); Performance (cs.PF) |
| Cite as: | arXiv:2607.24762 [cs.AI] |
| (or arXiv:2607.24762v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.24762 arXiv-issued DOI via DataCite |
Submission history
From: Dhravid Kumar [view email]
[v1]
Tue, 2 Jun 2026 14:16:58 UTC (14,508 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
HOBA (Hierarchical On-policy Bidding Agents) is a novel hierarchical reinforcement learning framework that enhances online advertising bidding systems by improving adaptability and reducing hyperparameter tuning costs. It utilizes a for hyperparameter inference, a SARSA agent for expert model selection, and a dynamic expert pool for bid execution, achieving a +3.6% increase in target cost during large-scale deployment and outperforming state-of-the-art baselines on AuctionNet.