SalArt-VQA: Diagnosing Whether VLMs Understand Salient Artifacts in Generated Images
Quick Answer
SalArt-VQA is a new benchmark for assessing vision-language models (VLMs) in detecting artifacts in AI-generated images.
Quick Take
SalArt-VQA is a new benchmark for assessing vision-language models (VLMs) in detecting artifacts in AI-generated images. Despite a 99.37% detection recall, the top-performing model only correctly answered 53.26% of artifact-related questions, highlighting a significant gap between detection accuracy and grounded understanding of artifacts.
Key Points
- SalArt-VQA includes 950 images and 3,681 multiple-choice questions.
- Four question types assess presence detection, localization, grounding, and defect identification.
- Top VLM model shows 99.37% recall but only 53.26% accuracy on artifact-related questions.
- Sensitive models often make unsupported claims about artifacts.
- Benchmark reveals hidden failure modes in VLMs' artifact understanding.
Paper Resources
Article Content
From source RSS / original summaryarXiv:2606. 12671v1 Announce Type: new Abstract: Vision-language models (VLMs) are increasingly used to detect whether AI-generated images contain visible artifacts, yet their ability to analyze such artifacts remains poorly understood. A correct image-level decision can still hide important failures: a model may correctly flag an artifact while relying on the wrong visual cue, selecting the wrong region, or describing a defect that the image does not support.
To evaluate these behaviors directly, we introduce SalArt-VQA, a diagnostic benchmark for fine-grained SALient ARTifact understanding in AI-generated images. SalArt-VQA contains 950 images and 3,681 human-authored multiple-choice questions spanning artifact images, matched real reference images, and paired generated reference images.
Four aligned question types evaluate presence detection, semantic localization, spatial grounding, and evidence-grounded defect identification, while the reference splits test calibration and abstention when the annotated defect is absent. Across 20 VLMs, SalArt-VQA reveals failures that image-level detection accuracy hides: the strongest model reaches 99. 37% detection recall on artifact images but answers all four artifact-side questions correctly on only 53. 26% of images.
Comparing artifact images with artifact-free references reveals a sensitivity-calibration tradeoff: sensitive models often make unsupported artifact claims, while conservative models avoid false alarms largely by missing real artifacts. These results show that high artifact detection accuracy alone does not imply grounded artifact understanding. SalArt-VQA exposes these hidden failure modes and provides a fine-grained evaluation of whether VLM artifact claims are supported by local visual evidence.
Reader Mode unavailable (could not extract clean content).
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →LLM-Guided ANN Index Optimization for Human-Object Interaction Retrieval
A phase-aware LLM agent optimizes human-object interaction retrieval, outperforming Optuna TPE by 33.3% and VDTuner by 34.2% on the HICO-DET benchmark. This method enhances throughput by 15.3x over UniIR and demonstrates strong transferability across vector database management systems.