ToolSense: A Diagnostic Framework for Auditing Parametric Tool Knowledge in LLMs
Quick Answer
ToolSense is an open-source diagnostic framework that evaluates parametric tool retrieval in LLMs, revealing a 50-64 percentage point drop in performance on realistic queries compared to standard benchmarks.
Quick Take
This indicates a significant knowledge-retrieval dissociation, as some models perform poorly on factual probes despite strong retrieval scores. The framework is available at https://github.com/SAP/toolsense.
Key Points
- ToolSense generates three benchmarks for tool retrieval evaluation: RRB, MCQ, and QA.
- Performance on RRB queries dropped significantly, revealing knowledge-retrieval dissociation.
- Five parametric model configurations were tested against ~47k tools in ToolBench.
- Despite high retrieval scores, some models scored near-random on factual probes.
- ToolSense framework and benchmarks are open-sourced for public use.
Paper Resources
Article Content
From source RSS / original summaryarXiv:2606. 12451v1 Announce Type: new Abstract: deployed as agents over large tool catalogs face a critical tool-retrieval bottleneck.
As embedding-based retrieval approaches rely on compact encoders that may under-capture specialized tool semantics, parametric tool retrieval addresses this by encoding each tool as a virtual token appended to the LLM vocabulary, fine-tuned in two stages (memorization then retrieval SFT) to use the LLM as a retriever, achieving strong performance on standard ToolBench retrieval benchmarks.
Yet these benchmarks use verbose, fully-specified queries, and their evaluation applies constrained decoding that restricts outputs to valid token paths, neither reveals whether the model actually understands its tools. We introduce \textbf{ToolSense}, an open-source LLM-powered diagnostic framework that takes any tool catalog as input and automatically generates three benchmarks: a Realistic Retrieval Benchmark (RRB) with queries at three ambiguity tiers, an MCQ probing benchmark, and a QA probing benchmark.
Applying ToolSense to ToolBench (~47k tools) and evaluating five parametric model training configurations reveals a knowledge-retrieval dissociation: on RRB queries, several configurations collapse by ~50-64 percentage points compared to fully-specified ToolBench benchmarks, falling below the embedding-model baseline. Additionally, despite strong retrieval performance, some models score near-random on factual probes, suggesting a knowledge-retrieval dissociation.
We open-source the ToolSense framework and the ToolBench diagnostic benchmarks at https://github. com/SAP/toolsense.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
HOBA (Hierarchical On-policy Bidding Agents) is a novel hierarchical reinforcement learning framework that enhances online advertising bidding systems by improving adaptability and reducing hyperparameter tuning costs. It utilizes a for hyperparameter inference, a SARSA agent for expert model selection, and a dynamic expert pool for bid execution, achieving a +3.6% increase in target cost during large-scale deployment and outperforming state-of-the-art baselines on AuctionNet.