Automatic Hard Example Synthesis with Multi-Level Agentic Data Curation
Quick Answer
This paper introduces an automated red-teaming framework for Multimodal Large Language Models (MLLMs) that synthesizes hard examples to enhance content safety.
Quick Take
By leveraging a , the approach reduces the False Negative Rate from 41.2% to 24.5% in a public image safety benchmark without human labeling, addressing vulnerabilities to adversarial attacks.
Key Points
- Automated framework synthesizes hard examples for MLLMs using multi-agent architecture.
- Reduces False Negative Rate from 41.2% to 24.5% in image safety benchmarks.
- No human intervention required for example generation and testing.
- Framework includes high-reasoning Architect agent and advanced image generator.
- Addresses vulnerabilities in content safety and moderation tasks.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Multimodal Large Language Models (MLLMs) are increasingly deployed for nuanced content safety and moderation tasks, yet they remain vulnerable to adversarial attacks and out-of-distribution edge cases. Traditional active learning and manual annotation fail to scale against the complexity and volume of novel multimodal threats. In this paper, we propose an automated, agentic red-teaming framework that systematically synthesizes difficult examples using an iterative strategy that proposes novel hypotheses as well as mutating on past attempts. Leveraging a multi-agent architecture that consists of a high-reasoning Architect agent, an advanced image generator, and a multi-level verification committee of LLM raters, our system autonomously uncovers boundary-pushing violations and ambiguous policy edge cases without any human intervention. By employing these carefully synthesized adversarial examples as in-context demonstrations via test-time Retrieval, we substantially improve the target model's robustness, reducing the False Negative Rate (FNR) from 41.2% to 24.5% in a public image safety benchmark without relying on any human labeling.
| Comments: | 23 pages; work in progress |
| Subjects: | Artificial Intelligence (cs.AI); Multiagent Systems (cs.MA) |
| Cite as: | arXiv:2607.14256 [cs.AI] |
| (or arXiv:2607.14256v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.14256 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Genglin Liu [view email]
[v1]
Wed, 15 Jul 2026 18:13:05 UTC (9,713 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
HOBA (Hierarchical On-policy Bidding Agents) is a novel hierarchical reinforcement learning framework that enhances online advertising bidding systems by improving adaptability and reducing hyperparameter tuning costs. It utilizes a for hyperparameter inference, a SARSA agent for expert model selection, and a dynamic expert pool for bid execution, achieving a +3.6% increase in target cost during large-scale deployment and outperforming state-of-the-art baselines on AuctionNet.