A Few Neurons Reveal When LLMs Misuse Tools: Sparse Detection and Selective Steering for Reliable Tool Use
Quick Answer
The PRISMS framework enhances tool-use reliability in LLMs like Qwen3, Llama, and Gemma by detecting failures with 1-2 MLP neurons, achieving up to 80% reduction in over-calling and a 14.2% increase in accuracy.
Quick Take
This lightweight approach allows for selective intervention, improving performance while minimizing collateral effects.
Key Points
- PRISMS detects failures in with high accuracy (ROC-AUC 0.90-1.00).
- Only 1-2 MLP neurons are needed for detecting missing calls, significantly reducing complexity.
- Over-calling rate decreased by 80%, improving tool-required accuracy by 14.2 percentage points.
- The framework supports bidirectional control over tool-calling behavior.
- Sparse detectors outperform dense baselines using 23-627 times fewer features.
DeepSignal Analysis
What happened
The PRISMS framework has been developed to enhance the reliability of tool use in large language models (LLMs) such as Qwen3, Llama, and Gemma. It utilizes a small number of MLP neurons to detect specific tool-use failures, achieving significant reductions in over-calling and improvements in accuracy.
Key evidence
- PRISMS detects over-calling and missing tool calls with ROC-AUC scores ranging from 0.90 to 1.00 across six models.
- The framework reduces the pooled over-calling rate by 80%, decreasing it from 0.131 to 0.026.
- Tool-required accuracy improves by 14.2 percentage points, increasing from 0.689 to 0.831.
Why it matters
The ability to detect and selectively intervene in tool-use failures is crucial for improving the performance of LLMs. By minimizing unnecessary calls and ensuring that necessary calls are made, PRISMS enhances the overall reliability of these models. This could lead to more effective applications in various domains where LLMs are deployed.
What to watch
Future developments should focus on the scalability of the PRISMS framework across different LLM architectures and its effectiveness in real-world applications. Additionally, monitoring the potential trade-offs between intervention and model performance will be important.
Paper Resources
Source Excerpt
Agentic exhibit three consequential failures: invalid arguments (validity), unnecessary calls (over-calling), and omitted calls when tools are needed (missing). We find that a small, failure-specific set of MLP neurons could distinguish such failures with linearly separable decision boundaries. Building on this observation, we introduce PRISMS (Probing Representations In Support of Monitoring and Steering), a closed-loop framework that shares a failure-specific neuron basis between
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.