PerceptionBench: Evaluating Atomic Visual Perception in Multimodal Large Language Models
Quick Answer
PerceptionBench introduces a novel benchmark to evaluate atomic visual perception in Multimodal Large Language Models (MLLMs), revealing that no model achieves over 60% accuracy in isolated perceptual tasks.
Quick Take
The study identifies ten atomic perceptual capabilities and constructs 3,000 targeted questions, highlighting significant gaps in current MLLM performance, particularly in perception-related hallucinations.
Key Points
- PerceptionBench evaluates atomic visual perception capabilities of MLLMs across 42 benchmarks.
- Constructed an error taxonomy defining ten atomic perceptual capabilities.
- 3,000 verified questions isolate single capabilities, focusing on perception over reasoning.
- No MLLM reached 60% accuracy, with perception-related hallucination being the weakest area.
- Provides a capability-level standard for diagnosing visual perception in MLLMs.
DeepSignal Analysis
What happened
PerceptionBench is a new benchmark aimed at evaluating atomic visual perception in Multimodal Large Language Models (MLLMs). The study found that no MLLM achieved over 60% accuracy in isolated perceptual tasks, highlighting significant performance gaps, especially in perception-related hallucinations.
Key evidence
- PerceptionBench identifies ten atomic perceptual capabilities and constructs 3,000 targeted questions to evaluate these capabilities.
- The benchmark results show that no model reaches 60% accuracy, indicating that atomic perception remains largely unresolved.
- Perception-related hallucination is identified as the weakest capability on average among the evaluated MLLMs.
Why it matters
The introduction of PerceptionBench addresses the limitations of existing benchmarks that conflate perceptual errors with reasoning failures. By isolating perceptual capabilities, it provides a clearer understanding of MLLMs' limitations in visual perception, which is crucial for advancing the field.
What to watch
Paper Resources
Source Excerpt
We introduce PerceptionBench, a benchmark specifically designed to evaluate the atomic visual perception capabilities of Multimodal (MLLMs). Existing benchmarks often fail to isolate perception: holistic evaluations conflate perceptual errors with failures in reasoning or domain knowledge, while application-driven benchmarks only cover narrow, fragmented domains shaped by heuristic designs. To address these limitations, PerceptionBench adopts a bottom-up approach: by diagno
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →ProMoE-FL: Prototype-conditioned Mixture of Experts for Multimodal Federated Learning with Missing Modalities
ProMoE-FL introduces a Prototype-conditioned Mixture-of-Experts framework for multimodal federated learning, effectively addressing missing modalities. It outperforms existing methods on four chest X-ray datasets, demonstrating superior feature synthesis capabilities in both homogeneous and heterogeneous settings.