BilliardPhys-Bench: Benchmarking Physical Reasoning and Visual Dynamics of Multimodal LLMs
Quick Answer
BilliardPhys-Bench introduces a benchmark for evaluating physical reasoning in multimodal LLMs, revealing significant performance drops in models like GPT, Claude, and Gemini as simulation complexity increases.
Quick Take
BilliardPhys-Bench introduces a benchmark for evaluating physical reasoning in multimodal LLMs, revealing significant performance drops in models like GPT, Claude, and Gemini as simulation complexity increases. A notable failure mode, termed 'stasis bias,' indicates models often predict no interaction when outcomes are less clear, highlighting the need for improved physical reasoning capabilities.
Key Points
- BilliardPhys-Bench tests physical reasoning in synthetic billiards environments.
- Evaluates models on predicting ball collisions, wall bounces, and final positions.
- Performance declines with increased simulation time and scene complexity.
- Models exhibit 'stasis bias,' predicting no interaction in ambiguous scenarios.
- Findings indicate a need for better physical inductive biases in multimodal architectures.
Paper Resources
Article Excerpt
From source RSS / original summaryarXiv:2605. 30900v1 Announce Type: new Abstract: Current multimodal models handle static image recognition well, but intuitive physical reasoning remains a weakness. Predicting how objects will move and interact from a single image is still difficult for these systems. We present BilliardPhys-Bench, a benchmark for physical reasoning in synthetic billiards environments. Its procedural engine generates randomized scenarios with friction and elastic collisions.
The benchmark tests three abilities: (1) predicting ball-to-ball collisions, (2) reasoning about wall bounces, and (3) estimating final ball positions after motion stops. We evaluate recent MLLMs from the GPT, Claude, Gemini, and Qwen families. Performance drops as simulation time increases and scene geometry grows more complex. We also observe a consistent failure mode we call "stasis bias": when the correct physical outcome is harder to infer, models tend to predict no interaction.
These findings show where current MLLMs break down on visual dynamics and point toward the need for better physical inductive biases in multimodal architectures.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →Automatic Ordinary Differential Equations Discovery For Biological Systems Using Large Language Model Powered Agentic System
The MEDA system utilizes large language models and symbolic regression to autonomously discover ordinary differential equations for biological systems, achieving strong structural recovery and biologically plausible models. It outperforms existing methods by integrating domain knowledge and mechanistic constraints, demonstrating effective retrieval and extrapolation capabilities.