ForecastBench-Sim: A Simulated-World Forecasting Benchmark
Quick Answer
ForecastBench-Sim introduces a simulated-world forecasting benchmark using Freeciv game rollouts, enabling continuous and binary forecasting tasks.
Quick Take
It allows for controlled evaluation of probabilistic reasoning in dynamic environments, addressing limitations of real-world benchmarks.
Key Points
- Utilizes game rollouts from Freeciv for benchmarking forecasting models.
- Enables continuous and binary forecasting questions across arbitrary time horizons.
- Facilitates scoring of counterfactual and causal questions in a simulated environment.
- Provides immediate resolution of rare or disruptive outcome examples.
- Aims to enhance understanding of probabilistic reasoning under dynamic conditions.
Paper Resources
Source Excerpt
Forecasting benchmarks for general-purpose AI systems usually inherit the constraints of the real world: outcomes resolve slowly, tail events are rare, and counterfactual questions are difficult to score. We introduce ForecastBench-Sim, a simulated-world forecasting benchmark built on game rollouts from Freeciv, a turn-based strategy game modelled on the Civilization series. Forecasters receive a fixed world report (a structured snapshot of the current game state) and answer questions about hidd
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
HOBA (Hierarchical On-policy Bidding Agents) is a novel hierarchical reinforcement learning framework that enhances online advertising bidding systems by improving adaptability and reducing hyperparameter tuning costs. It utilizes a for hyperparameter inference, a SARSA agent for expert model selection, and a dynamic expert pool for bid execution, achieving a +3.6% increase in target cost during large-scale deployment and outperforming state-of-the-art baselines on AuctionNet.