FAGER: Factually Grounded Evaluation and Refinement of Text-to-Image Models
Quick Take
FAGER enhances text-to-image evaluation by assessing factual correctness and providing actionable feedback.
Key Points
- Evaluates images based on visually verifiable facts.
- Introduces a structured factual rubric for assessment.
- Outperforms existing metrics across multiple datasets.
📖 Reader Mode
~2 min readAbstract:Existing text-to-image (T2I) evaluation metrics mainly assess whether generated images align with information explicitly stated in the prompt, but often fail to capture factual requirements that are implicit, externally grounded, or identity-defining. As a result, they are not well suited for evaluating factual correctness in prompts involving scientific knowledge, historical facts, products, or culture-specific concepts. We propose FActually Grounded Evaluation and Refinement (FAGER), an agentic framework that evaluates whether generated images correctly reflect visually verifiable facts grounded in or implied by the prompt, while also providing actionable feedback for improvement. FAGER first constructs a structured factual rubric by combining LLM-based fact proposal with reference-guided visual fact extraction and verification, then converts the rubric into question-answer pairs for VLM-based evaluation. To validate FAGER as a factuality metric, we introduce a Factual A/B test, which measures whether a metric prefers factual reference images over corresponding generated images. Across five datasets spanning science, history, products, culture, and knowledge-intensive concepts, FAGER consistently outperforms prior metrics on this test. We further show that FAGER can be used to refine T2I outputs in a fully training-free manner, yielding substantial factuality gains across datasets.
| Comments: | It was accepted for an oral presentation at the 2nd Workshop on the Evaluation of Generative Foundation Models (EVGENFM2026) at CVPR 2026. Total 8 pages (1 page for references). 5 figures |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2605.19111 [cs.CV] |
| (or arXiv:2605.19111v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2605.19111 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Youngsun Lim [view email]
[v1]
Mon, 18 May 2026 20:57:07 UTC (13,030 KB)
— Originally published at arxiv.org
More from arXiv cs.CV
See more →arXiv cs.CV·Jinhao Jing, Zheng Ma, Jinwei Liang, Qiannian Zhao, Shawn Chen, Jing Yang, Por Lip Yee, Prayag Tiwari, Jingjing Bai, Benyou Wang, Lewei Lu, Zhan Su
1d ago FeaturedOriginal
GeoSym127K: Scalable Symbolically-verifiable Synthesis for Multimodal Geometric Reasoning
AI Summary
GeoSym127K introduces a scalable neuro-symbolic framework for enhanced geometric reasoning in multimodal models.