When Search Agents Should Ask: DiscoBench for Clarification-Aware Deep Search
Quick Answer
DiscoBench introduces a benchmark for clarification-aware deep search, assessing LLMs' ability to detect ambiguity and ask clarifying questions.
Quick Take
DiscoBench introduces a benchmark for clarification-aware deep search, assessing LLMs' ability to detect ambiguity and ask clarifying questions. Experiments reveal that ambiguity detection and clarification are distinct capabilities, with repeated searches often performing worse than direct guessing. This highlights a significant gap in current search agents' interactive problem-solving abilities.
Key Points
- DiscoBench includes 211 samples and 463 ambiguity instances across 11 domains.
- The benchmark evaluates task utility, ambiguity detection, interaction strategy, and cost efficiency.
- Experiments show that LLMs struggle with ambiguity and clarification, impacting search accuracy.
- Repeated searches without clarification often yield worse results than direct guessing.
- The study identifies a critical gap in interactive problem-solving for search agents.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Search agents powered by large language models (LLMs) are increasingly used to solve complex information-seeking tasks, requiring multi-step retrieval and reasoning to fulfill user goals. However, existing benchmarks often assume that user queries are complete and explicit, overlooking the fact that real-world search requests are frequently vague, underspecified, or even factually incorrect. In deep search scenarios, such ambiguity can propagate along multi-step reasoning chains and lead agents toward incorrect search trajectories. To address this gap, we introduce DiscoBench, a benchmark for clarification-aware deep search, designed to evaluate whether search agents can proactively identify ambiguity, ask effective clarification questions, and recover correct reasoning paths through user interaction. DiscoBench contains 211 samples and 463 ambiguity instances across 11 real-world domains, covering four ambiguity types. We further design a user simulator for multi-turn interaction and evaluate model performance from four perspectives: task utility, ambiguity detection, interaction strategy, and cost efficiency. Experiments on representative LLMs show that ambiguity detection and effective clarification are distinct capabilities, and that repeatedly searching instead of asking for clarification often performs worse than direct guessing, highlighting a critical gap between retrieval ability and interactive problem-solving in current search agents.
| Comments: | 26 pages, 7 figures, 12 tables |
| Subjects: | Computation and Language (cs.CL) |
| ACM classes: | I.2.7 |
| Cite as: | arXiv:2606.27669 [cs.CL] |
| (or arXiv:2606.27669v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2606.27669 arXiv-issued DOI via DataCite |
Submission history
From: Zhu Zhihao [view email]
[v1]
Fri, 26 Jun 2026 02:57:15 UTC (8,276 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Quantifying Prior Dominance in Systems
The study introduces the Normalized Context Utilization (NCU) metric to evaluate Retrieval-Augmented Generation (RAG) systems, revealing that Small Language Models (SLMs) outperform larger models in factual extraction. The findings indicate that traditional scaling laws yield diminishing returns, with a commercial API frequently failing against adversarial evidence due to systemic confidence collapse.