Step-by-Step Optimization-like Reasoning in LLMs over Expanding Search Spaces
Quick Answer
The paper introduces OPT*, a scalable framework for training LLMs in step-by-step optimization-like reasoning, enhancing decision-making in complex search spaces.
Quick Take
It evaluates two regimes: solver-guided online policy optimization and search-based offline RL, demonstrating improved reasoning capabilities without new human labels.
Key Points
- OPT* provides feasibility checkers and evaluators for scalable optimization tasks.
- The framework expands search spaces using a complexity parameter without new labels.
- Two regimes are explored: solver-guided optimization and search-based offline RL.
- Empirical results show training on OPT* enhances step-by-step reasoning efficiency.
- Success in large search spaces relates to information extracted per search budget.
Paper Resources
Article Excerpt
From source RSS / original summaryarXiv:2606. 05464v1 Announce Type: new Abstract: Verifiable reward training has improved mathematical and coding reasoning, but these domains capture only part of step-by-step decision making. Many real-world tasks require finding a high-value feasible plan among many valid alternatives.
We introduce OPT*, a scalable family of optimization-style tasks for training and evaluating step-by-step optimization-like reasoning along a complexity axis: each task provides a feasibility checker and evaluator, while a complexity parameter expands the search space without requiring new human labels.
This motivates studying these tasks in two regimes: (i) solver-guided online policy optimization, which uses a solver as a value oracle for partial states and applies rank-based reward shaping to reinforce better next steps, and (ii) search-based offline RL when such solvers are unavailable. Theoretically, we relate success in large search spaces to the information a reasoner extracts per unit of search budget.
Empirically, we ablate the ingredients that make search efficient on OPT* and show that training on OPT* improves step-by-step optimization-like reasoning.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
HOBA (Hierarchical On-policy Bidding Agents) is a novel hierarchical reinforcement learning framework that enhances online advertising bidding systems by improving adaptability and reducing hyperparameter tuning costs. It utilizes a for hyperparameter inference, a SARSA agent for expert model selection, and a dynamic expert pool for bid execution, achieving a +3.6% increase in target cost during large-scale deployment and outperforming state-of-the-art baselines on AuctionNet.