Automatic Hard Example Synthesis with Multi-Level Agentic Data Curation
Quick Answer
This paper introduces an automated red-teaming framework for Multimodal Large Language Models (MLLMs) that synthesizes hard examples to enhance content safety.
Quick Take
This paper introduces an automated red-teaming framework for Multimodal Large Language Models (MLLMs) that synthesizes hard examples to enhance content safety. By leveraging a , the approach reduces the False Negative Rate from 41.2% to 24.5% in a public image safety benchmark without human labeling, addressing vulnerabilities to adversarial attacks.
Key Points
- Automated framework synthesizes hard examples for MLLMs using multi-agent architecture.
- Reduces False Negative Rate from 41.2% to 24.5% in image safety benchmarks.
- No human intervention required for example generation and testing.
- Framework includes high-reasoning Architect agent and advanced image generator.
- Addresses vulnerabilities in content safety and moderation tasks.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Multimodal Large Language Models (MLLMs) are increasingly deployed for nuanced content safety and moderation tasks, yet they remain vulnerable to adversarial attacks and out-of-distribution edge cases. Traditional active learning and manual annotation fail to scale against the complexity and volume of novel multimodal threats. In this paper, we propose an automated, agentic red-teaming framework that systematically synthesizes difficult examples using an iterative strategy that proposes novel hypotheses as well as mutating on past attempts. Leveraging a multi-agent architecture that consists of a high-reasoning Architect agent, an advanced image generator, and a multi-level verification committee of LLM raters, our system autonomously uncovers boundary-pushing violations and ambiguous policy edge cases without any human intervention. By employing these carefully synthesized adversarial examples as in-context demonstrations via test-time Retrieval, we substantially improve the target model's robustness, reducing the False Negative Rate (FNR) from 41.2% to 24.5% in a public image safety benchmark without relying on any human labeling.
| Comments: | 23 pages; work in progress |
| Subjects: | Artificial Intelligence (cs.AI); Multiagent Systems (cs.MA) |
| Cite as: | arXiv:2607.14256 [cs.AI] |
| (or arXiv:2607.14256v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.14256 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Genglin Liu [view email]
[v1]
Wed, 15 Jul 2026 18:13:05 UTC (9,713 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →Automatic Ordinary Differential Equations Discovery For Biological Systems Using Large Language Model Powered Agentic System
The MEDA system utilizes large language models and symbolic regression to autonomously discover ordinary differential equations for biological systems, achieving strong structural recovery and biologically plausible models. It outperforms existing methods by integrating domain knowledge and mechanistic constraints, demonstrating effective retrieval and extrapolation capabilities.