BilliardPhys-Bench: Benchmarking Physical Reasoning and Visual Dynamics of Multimodal LLMs
Quick Answer
BilliardPhys-Bench introduces a benchmark for evaluating physical reasoning in multimodal LLMs, revealing significant performance drops in models like GPT, Claude, and Gemini as simulation complexity increases.
Quick Take
A notable failure mode, termed 'stasis bias,' indicates models often predict no interaction when outcomes are less clear, highlighting the need for improved physical reasoning capabilities.
Key Points
- BilliardPhys-Bench tests physical reasoning in synthetic billiards environments.
- Evaluates models on predicting ball collisions, wall bounces, and final positions.
- Performance declines with increased simulation time and scene complexity.
- Models exhibit 'stasis bias,' predicting no interaction in ambiguous scenarios.
- Findings indicate a need for better physical inductive biases in multimodal architectures.
Paper Resources
Article Excerpt
From source RSS / original summaryarXiv:2605. 30900v1 Announce Type: new Abstract: Current multimodal models handle static image recognition well, but intuitive physical reasoning remains a weakness. Predicting how objects will move and interact from a single image is still difficult for these systems. We present BilliardPhys-Bench, a benchmark for physical reasoning in synthetic billiards environments. Its procedural engine generates randomized scenarios with friction and elastic collisions.
The benchmark tests three abilities: (1) predicting ball-to-ball collisions, (2) reasoning about wall bounces, and (3) estimating final ball positions after motion stops. We evaluate recent MLLMs from the GPT, Claude, Gemini, and Qwen families. Performance drops as simulation time increases and scene geometry grows more complex. We also observe a consistent failure mode we call "stasis bias": when the correct physical outcome is harder to infer, models tend to predict no interaction.
These findings show where current MLLMs break down on visual dynamics and point toward the need for better physical inductive biases in multimodal architectures.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
HOBA (Hierarchical On-policy Bidding Agents) is a novel hierarchical reinforcement learning framework that enhances online advertising bidding systems by improving adaptability and reducing hyperparameter tuning costs. It utilizes a for hyperparameter inference, a SARSA agent for expert model selection, and a dynamic expert pool for bid execution, achieving a +3.6% increase in target cost during large-scale deployment and outperforming state-of-the-art baselines on AuctionNet.