Diagnosing Tool-Selection Reasoning in LLM Agents with Canary Tools
Quick Answer
The study introduces 'canary tools' to diagnose tool-selection weaknesses in LLMs, revealing that model capability significantly affects susceptibility to errors.
Quick Take
Evaluating eight models, including Claude Opus 4.8 and Llama 3.1, it finds a 36x variance in susceptibility rates, with capability mirages particularly trapping frontier models. The framework and findings are released for further research.
Key Points
- Canary tools diagnose specific tool-selection weaknesses in agents.
- Susceptibility rates vary 36x across models, highest in Llama 3.1.
- Capability mirages trap frontier models, while other types affect smaller models.
- Task success correlates with susceptibility, indicating reasoning measurement.
- Framework, schemas, tasks, and logs are publicly released for research.
DeepSignal Analysis
What happened
The study introduces 'canary tools' to diagnose specific weaknesses in tool selection by large language models (LLMs). Evaluating eight models, including Claude Opus 4.8 and Llama 3.1, it finds a significant variance in susceptibility rates, indicating that model capability influences error rates. The findings and framework are made available for further research.
Key evidence
- The study evaluated eight models across three capability tiers on 120 tasks, resulting in 8,640 runs and a 2,880-run subtlety ablation.
- The susceptibility rate varied by 36 times among models, with Claude Opus 4.8 showing the lowest and Llama 3.1 8B the highest rates.
- Capability mirages predominantly affected frontier models, while other types of tool-selection weaknesses were less impactful on stronger models.
Why it matters
Understanding the reasoning behind tool selection in LLMs is crucial for improving their reliability and effectiveness. The findings suggest that enhancing model capabilities can significantly reduce error rates. Moreover, the taxonomy of weaknesses provides a structured approach to diagnosing and addressing specific issues in model performance, which can inform future developments in AI.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Agent evaluations tell us that a model picked the wrong tool, but rarely why. We introduce canary tools: diagnostic probe tools planted in an agent's Model Context Protocol (MCP) tool set, each engineered to probe one specific tool-selection weakness. A six-type taxonomy (semantic decoys, parameter traps, capability mirages, prerequisite blindness, temporal decoys, and granularity traps) turns a single "wrong tool" outcome into a multi-dimensional profile of how a model reasons about tools. We evaluate eight models -- six hosted and two 8B open-weight -- spanning three capability tiers, on 120 tasks across three canary-density conditions and three seeds (8,640 runs), plus a 2,880-run subtlety ablation. Task success is graded by a provider-independent judge, corroborated by a second independent judge (Cohen's kappa = 0.75). We report three findings. First, susceptibility drops sharply as models get more capable: the per-task canary susceptibility rate (CSR) ranges about 36x across models, lowest for Claude Opus 4.8 and highest for Llama 3.1 8B. Second, capability tier alone does not predict safety: the most susceptible hosted model is mid-tier, and within a provider the cheaper model can be the safer one. Third, the taxonomy is capability-stratified: capability mirages most reliably trap frontier models, while the other types are largely inert on strong models but fire on small open models, so they discriminate by capability rather than being weak. Softening each canary's give-away phrase leaves frontier CSR essentially unchanged, evidence that the probes measure reasoning, not phrase-spotting. Susceptibility also predicts task failure (Spearman rho = -0.34), while the most robust models are not significantly degraded by canary pressure. We release the framework, canary schemas, tasks, and logs.
| Comments: | 10 pages, 9 figures, 5 tables |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2608.04719 [cs.AI] |
| (or arXiv:2608.04719v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2608.04719 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Atul Anand [view email]
[v1]
Wed, 5 Aug 2026 11:38:33 UTC (641 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
HOBA (Hierarchical On-policy Bidding Agents) is a novel hierarchical reinforcement learning framework that enhances online advertising bidding systems by improving adaptability and reducing hyperparameter tuning costs. It utilizes a for hyperparameter inference, a SARSA agent for expert model selection, and a dynamic expert pool for bid execution, achieving a +3.6% increase in target cost during large-scale deployment and outperforming state-of-the-art baselines on AuctionNet.