FindIt: A Format-Informed Visual Detection Benchmark for Generalist Multimodal LLMs
Quick Take
The study introduces FindIt, a benchmark for evaluating the localization capabilities of multimodal large language models (MLLMs) across four tasks: object detection, referring expression detection, instance-level detection, and video-based detection. It highlights the sensitivity of current models to formatting constraints, revealing significant performance limitations in practical applications.
Key Points
- FindIt benchmark evaluates MLLMs on structured localization tasks.
- Four core task categories include object detection and video-based detection.
- Current models struggle with formatting constraints, impacting performance.
- Benchmark provides insights into strengths and weaknesses of MLLMs.
- Study aims to improve multimodal model design and evaluation.
Article Content
From source RSS / original summaryarXiv:2606. 04282v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) are predominantly evaluated on free-form vision-language tasks such as visual question answering, captioning, and summarization. However, their practical use is rapidly expanding to more structured computer vision settings, where users prompt models to perform localization-centric tasks such as object detection, often within larger agentic or decision-making systems.
Despite this shift, there is currently no standardized benchmark that systematically evaluates these capabilities at scale. In this work, we introduce the first comprehensive benchmark specifically designed to assess the promptable localization abilities of generalist MLLMs. Our benchmark spans four core task categories: object detection, referring expression detection, instance-level detection, and video-based detection.
To enable consistent and fair evaluation, we develop a unified framework that standardizes inputs, enforces parsable bounding box outputs, and defines transparent evaluation protocols across tasks. Using this suite, we evaluate a diverse set of open-source and proprietary MLLMs, providing an in-depth analysis of their performance and limitations.
Beyond accuracy, we examine models' ability to adhere to output format specifications, showing that current systems are highly sensitive to formatting constraints and often fail to generalize even to minor variations. Our results highlight both the strengths and shortcomings of state-of-the-art MLLMs in localization settings, and point toward important directions for improving multimodal model design and evaluation.
Reader Mode unavailable (could not extract clean content).
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →Optimal Transport Flow Matching by Design
The study presents a novel approach to optimal transport (OT) flow matching, reformulating the problem by treating the prior as a design choice. This method achieves over 2x reduction in trajectory curvature compared to existing methods, improving generation quality in few-step regimes without altering the flow model. The approach integrates seamlessly with latent-space models and classifier-free guidance.