Hallucination Self-Play: Bootstrapping Reinforced Detector via Evolved Generator
Quick Answer
This paper shows that The Hallucination Self-Play (HSP) framework enhances a detector for identifying hallucinations in LLM outputs by bootstrapping it with an evolved generator.
Quick Take
The Hallucination Self-Play (HSP) framework enhances a detector for identifying hallucinations in LLM outputs by bootstrapping it with an evolved generator. This approach, which utilizes reinforcement learning from AI feedback, allows a small LLM to match or outperform advanced models on the RAGTruth benchmark without external supervision.
Key Points
- HSP framework involves a detector and an evolved generator from the same base model.
- The detector is fine-tuned on human-labeled data and acts as a reward model.
- Reinforcement learning from AI feedback optimizes the generator's hallucination data synthesis.
- Experiments show significant performance improvements on the RAGTruth benchmark.
- The framework allows small LLMs to compete with advanced models effectively.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Identifying faithfulness hallucinations in LLM-generated outputs remains challenging due to the scarcity of high-quality annotated data. Recent work relies on advanced LLMs to synthesize training data, including rationales, labels, and hallucinated claims. However, these methods treat the generator as a static component, limiting iterative improvement of the detector. To address this limitation, we introduce Hallucination Self-Play (HSP), a novel framework that enables the detector to bootstrap with an evolved generator. HSP involves two roles initialized from the same base model, a detector that assesses the faithfulness of model outputs, and a generator that produces increasingly hard-to-detect hallucinated responses. Specifically, the detector is first fine-tuned on human-labeled data and then employed as a reward model to train the generator via reinforcement learning from AI feedback (RLAIF). In turn, the evolved generator synthesizes hallucination data to further optimize the detector through rule-based reinforcement learning. Experiments on RAGTruth benchmark and two model families demonstrate that the proposed framework can progressively enhance a small LLM to match or even outperform advanced LLMs without external supervision. Our code is available at this https URL .
| Comments: | Accepted to COLM 2026. Camera-ready version to appear |
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2607.07993 [cs.CL] |
| (or arXiv:2607.07993v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.07993 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Shiping Yang [view email]
[v1]
Wed, 8 Jul 2026 23:54:36 UTC (160 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Quantifying Prior Dominance in Systems
The study introduces the Normalized Context Utilization (NCU) metric to evaluate Retrieval-Augmented Generation (RAG) systems, revealing that Small Language Models (SLMs) outperform larger models in factual extraction. The findings indicate that traditional scaling laws yield diminishing returns, with a commercial API frequently failing against adversarial evidence due to systemic confidence collapse.