Insights Generator: Systematic Corpus-Level Trace Diagnostics for LLM Agents
Quick Answer
This paper shows that The Insights Generator (IG) enhances LLM agent diagnostics by automating hypothesis testing across execution trace corpora, improving scaffold performance by 30.4 percentage points.
Quick Take
IG's architecture provides evidence-backed insights, achieving detection coverage comparable to existing methods and receiving high ratings from domain experts for depth and quality.
Key Points
- IG automates the diagnosis of agent failures across large execution trace corpora.
- Human experts improved scaffold performance by 30.4 percentage points using IG insights.
- IG's scout-investigator architecture matches detection coverage of competing diagnostic methods.
- Domain experts rated IG reports highly for depth and evidence quality.
- The system produces grounded natural-language insights linked to supporting evidence.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Diagnosing failures in LLM agents remains largely manual. Practitioners inspect a small subset of execution traces, form ad-hoc hypotheses, and iterate. This process misses patterns that only emerge across trace populations and does not scale to production corpora where individual traces span tens of thousands of tokens. We formalize the problem of corpus-level trace diagnostics. Given a corpus of execution traces, the goal is to produce grounded natural-language insights that characterize systematic behavioral patterns across trace groups, each linked to supporting evidence. We present the Insights Generator (IG), a multi-agent system that answers diagnostic questions by proposing and testing hypotheses across the trace corpus to produce an evidence-backed insights report. We evaluate IG across qualitative and objective dimensions, spanning rubric-based report assessment and downstream performance improvements achieved by implementing IG insights. Human experts using IG reports improve scaffold performance by 30.4pp over the unmodified baseline scaffold, and coding agents leveraging IG-derived insights show consistent and stable gains. Across benchmarks, IG's scout-investigator architecture produces findings comparable in detection coverage to competing approaches, while domain experts rated IG reports as leading depth and evidence quality.
| Subjects: | Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Software Engineering (cs.SE) |
| Cite as: | arXiv:2605.21347 [cs.AI] |
| (or arXiv:2605.21347v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2605.21347 arXiv-issued DOI via DataCite |
Submission history
From: Veronica Chatrath [view email]
[v1]
Wed, 20 May 2026 16:13:53 UTC (1,429 KB)
[v2]
Thu, 21 May 2026 16:51:51 UTC (1,429 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
HOBA (Hierarchical On-policy Bidding Agents) is a novel hierarchical reinforcement learning framework that enhances online advertising bidding systems by improving adaptability and reducing hyperparameter tuning costs. It utilizes a for hyperparameter inference, a SARSA agent for expert model selection, and a dynamic expert pool for bid execution, achieving a +3.6% increase in target cost during large-scale deployment and outperforming state-of-the-art baselines on AuctionNet.