InferenceBench: A Benchmark for Open-Ended LLM Inference Optimization by AI Agents
Quick Answer
InferenceBench introduces a benchmark for optimizing LLM inference by AI agents, demonstrating improvements over a naive PyTorch baseline by up to 8.08x.
Quick Take
Agents often converge on a single framework and struggle with diverse configuration proposals, indicating a need for better exploration strategies in open-ended tasks.
Key Points
- InferenceBench evaluates AI agents on open-ended inference optimization.
- Agents improved performance by up to 8.08x over a naive PyTorch baseline.
- Four optimization scenarios target distinct inference bottlenecks.
- Agents predominantly converge on one inference framework during optimization.
- Exploration of diverse strategies is limited, affecting overall optimization potential.
DeepSignal Analysis
What happened
InferenceBench is a newly introduced benchmark aimed at optimizing LLM inference through AI agents. The benchmark shows that agents can improve performance over a naive PyTorch baseline by up to 8.08 times, although they often converge on a single framework and do not explore diverse configurations effectively.
Key evidence
- InferenceBench allows agents to optimize LLM inference speed using a target LLM, one H100 GPU, and a two-hour time budget.
- Agents improved performance over a naive PyTorch baseline by as much as 8.08 times, but still fell short of a simple hyperparameter search that achieved up to 11.53 times improvement.
- Qualitative analysis indicates that agents primarily converge on a single inference framework and spend time re-measuring rather than exploring diverse optimization strategies.
Why it matters
The findings from InferenceBench highlight the limitations of current AI agents in open-ended tasks, particularly their tendency to rely on memorized solutions rather than innovative strategies. This raises questions about the effectiveness of AI agents in complex optimization scenarios and suggests a need for improved exploration techniques.
Paper Resources
📖 Reader Mode
~2 min readAbstract:AI agents are increasingly used to automate research and development tasks, yet existing benchmarks typically evaluate them on prescribed workflows or narrow action spaces. Even nominally open-ended tasks can often be solved by retrieving a well-known recipe and tuning a few hyperparameters, making it unclear whether strong results reflect genuine optimization or memorized solutions. We introduce InferenceBench, where an agent must deploy an OpenAI-compatible inference server and optimize the speed of LLM inference. Each agent receives a target LLM, one H100 GPU, an optimization scenario, and a wall-clock time budget of two hours. Three optimization scenarios isolate distinct bottlenecks of inference (prefill latency, decode latency, and concurrent request throughput) and a fourth balances all three at the same time. Across 15 frontier agent configurations, agents reliably improve over a naive PyTorch baseline (up to $8.08\times$) and often match or exceed serving engines with default settings ($4.05\times$ for vLLM), but still fall below a simple hyperparameter search under the same time budget (up to $11.53\times$). Qualitative analysis of agent trajectories shows that although agents enumerate many relevant optimization techniques, they overwhelmingly converge on a single inference framework. They test only a few distinct configurations and spend the remaining budget re-measuring, repairing, or optimizing hyperparameters rather than exploring substantially different strategies. This suggests the bottleneck is not domain knowledge, but the ability to propose diverse configurations, evaluate them systematically, and submit the best identified solution. Overall, InferenceBench reflects the ability of agents to operate in an open-ended AI engineering setting, where memorized solutions lead to limited improvements.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.20468 [cs.AI] |
| (or arXiv:2607.20468v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.20468 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jehyeok Yeon [view email]
[v1]
Wed, 20 May 2026 15:55:21 UTC (1,018 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
HOBA (Hierarchical On-policy Bidding Agents) is a novel hierarchical reinforcement learning framework that enhances online advertising bidding systems by improving adaptability and reducing hyperparameter tuning costs. It utilizes a for hyperparameter inference, a SARSA agent for expert model selection, and a dynamic expert pool for bid execution, achieving a +3.6% increase in target cost during large-scale deployment and outperforming state-of-the-art baselines on AuctionNet.