BBOWP-Bench: Evaluating LLMs on Black-Box Optimization Word Problems
Quick Answer
The BBOWP-Bench introduces a novel framework for evaluating LLMs on black-box optimization problems, allowing systems to infer search spaces and algorithms from natural language.
Quick Take
Initial results show can select suitable algorithms but struggle with effective search space design, particularly in complex scenarios.
Key Points
- BBOWP-Bench establishes a dataset and evaluation framework for black-box optimization problems.
- LLMs can select suitable algorithms based on evaluation budgets but struggle with search space design.
- The benchmark includes natural-language problem descriptions and executable evaluation environments.
- Search space design issues arise when problem descriptions are less informative or highly specific.
DeepSignal Analysis
What happened
The BBOWP-Bench framework was introduced to evaluate large language models (LLMs) on black-box optimization problems. Initial evaluations indicate that while LLMs can select appropriate algorithms, they face challenges in designing effective search spaces, especially in complex scenarios.
Key evidence
- The BBOWP-Bench establishes a dataset and evaluation framework for black-box optimization problems, combining natural-language descriptions with executable environments.
- Current LLMs can select suitable algorithms based on evaluation budgets, but they struggle with search space design when problem descriptions are less informative.
- The benchmark highlights the importance of search space design and algorithm selection in solving black-box optimization problems, which are crucial for effective problem-solving.
Why it matters
This research is significant as it addresses the challenges of automating optimization problem formulation using LLMs, which could enhance the efficiency of solving complex optimization tasks. Understanding the limitations of LLMs in this context can guide future improvements in AI-driven optimization techniques.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Formulating an optimization problem strongly affects the quality of the final solution, yet good formulations usually require substantial expertise. Recent studies have therefore examined how to automatically derive optimization problems from natural-language descriptions, but existing benchmarks focus on settings where objectives and constraints can be written explicitly as mathematical expressions. Many practically important problems are naturally treated as black-box optimization (BBO) problems, in which only objective values are observable, and the functional form is unavailable. In BBO, the search space design, a part of the problem formulation, and the selection of the optimization algorithm are crucial for problem-solving. Automating these processes with large language models (LLMs) is a significant challenge. This paper introduces Black-Box Optimization Word Problems (BBOWP), a novel problem setting in which a system must infer both a search space and an optimization algorithm from a natural-language description of a black-box optimization task. To support research on this setting, we establish the BBOWP Benchmark Suite (BBOWP-Bench), a dataset and evaluation framework for BBOWP. Each instance combines a natural-language problem description, an executable evaluation environment, and a human-designed baseline formulation, allowing evaluation of both search-space design and algorithm selection. Using this benchmark, we provide the first evaluation of LLMs and show that current LLMs are capable of selecting suitable algorithms based on the given evaluation budget. However, they sometimes struggle with search space design, particularly in identifying important variables and balancing their ranges when the problem description is less informative or the search space is highly problem-specific. Our code and dataset are available at this https URL.
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG); Neural and Evolutionary Computing (cs.NE) |
| Cite as: | arXiv:2608.02612 [cs.CL] |
| (or arXiv:2608.02612v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2608.02612 arXiv-issued DOI via DataCite |
Submission history
From: Yutaro Yamada [view email]
[v1]
Wed, 20 May 2026 08:11:23 UTC (812 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.