DynaSchedBench: Calibrated Dynamic Scheduling Benchmarks and Observability Paradox in LLM-based Scheduling Agents
Quick Answer
DynaSchedBench introduces a calibrated framework for Dynamic Flexible Job Shop Scheduling, utilizing Sequential Event-Space Calibrator (SESC) to enhance efficiency and performance metrics.
Quick Take
It reveals an 'Observability Paradox' where excessive structural information can degrade -based scheduling agents' performance, highlighting their limitations against traditional dispatching methods.
Key Points
- DynaSchedBench controls instance generation for DFJSP, improving benchmarking accuracy.
- SESC computes a Schedule Stress Index, enhancing computational efficiency over evolutionary methods.
- LLM-based agents struggle with dynamic scheduling, often underperforming against heuristic baselines.
- Providing full structural information can paradoxically degrade agent performance.
- Tool-augmented strategies show limited improvement in scheduling outcomes.
Paper Resources
Article Content
From source RSS / original summaryarXiv:2605. 27566v1 Announce Type: new Abstract: Progress in neural combinatorial optimization for Dynamic Flexible Job Shop Scheduling Problem (DFJSP) is currently hindered by a methodological tension: static benchmarks encourage benchmark overfitting, while uncalibrated generators obscure algorithmic capability with stochastic noise. To resolve this, we introduce \textbf{DynaSchedBench}, a diagnostic framework for DFJSP that rigorously controls the instance-generation process.
Instead of relying on parameter sampling, our approach utilizes Sequential Event-Space Calibrator (SESC) that computes a novel Schedule Stress Index (SSI) to stratify instances by difficulty. We demonstrate that SESC is substantially more computationally efficient than evolutionary baselines while converging reliably to the target metrics.
The framework integrates modular components for instance generation, snapshot-based simulation, agents, evaluation, and visualization, thereby enabling rigorous testing of reactive and lookahead-based policies. Leveraging this calibrated environment, we identify key limitations of -based scheduling agents.
Specifically, in step-wise online decision-making for dynamic scheduling, we identify an ``Observability Paradox'': providing agents with oracle access to full structural information can degrade policy performance, underperforming concise information.
Furthermore, despite substantial token overhead, tool-augmented and refinement strategies fail to reliably improve performance, and most LLM agents fail to consistently surpass strong dispatching baselines-behaving more like robust heuristic approximators than superior optimizers.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
HOBA (Hierarchical On-policy Bidding Agents) is a novel hierarchical reinforcement learning framework that enhances online advertising bidding systems by improving adaptability and reducing hyperparameter tuning costs. It utilizes a for hyperparameter inference, a SARSA agent for expert model selection, and a dynamic expert pool for bid execution, achieving a +3.6% increase in target cost during large-scale deployment and outperforming state-of-the-art baselines on AuctionNet.