Safeguarding LLM Agents from Misalignment through Provenance Analysis
Quick Answer
This paper shows that ProvenanceGuard, a new framework for LLM agents, reduces misalignment error rates from 42.9% to 1.8% on Agent-SafetyBench and from 32.1% to 17.3% on WorkBench, enhancing alignment with user intent through structured provenance analysis.
Key Points
- ProvenanceGuard analyzes agent actions for misalignment before tool execution.
- Error rates on misaligned traces dropped significantly across multiple benchmarks.
- Intervention burden on successful task traces reduced from 30.5% to 12.8%.
- No significant increase in unnecessary interventions on aligned traces.
- Framework leverages provenance analysis for effective alignment detection.
Paper Resources
📖 Reader Mode
~2 min readAbstract:As LLM agents gain increasing access to powerful tools, ensuring that their actions are aligned with the user's intent becomes critical. When an agent's proposed tool invocation deviates from the user's intent -- a phenomenon called misalignment -- it may lead to harmful consequences that are difficult to undo. Existing runtime guardrails rely on an LLM-as-a-judge paradigm that lacks a systematic framework for reasoning about alignment, often producing judgments that are inconsistent or difficult to audit. Motivated by provenance analysis, we propose a provenance-based conceptual framework that formalizes misalignment detection as determining whether a proposed tool call is supported by traceable evidence in the agent's context. Building on this framework, we propose ProvenanceGuard, a multi-stage pipeline that analyzes the agent's action for three types of misalignment before the selected tool is executed and only allows the action to take place when it is considered aligned with the user's input query. We evaluated our proposed approach on two different benchmarks, Agent-SafetyBench and WorkBench, across 10 backbone LLMs. Compared to the LLM-as-a-judge baseline, ProvenanceGuard reduces error rate on misaligned traces from 42.9% to 1.8% on Agent-SafetyBench and from 32.1% to 17.3% on WorkBench, while reducing intervention burden on task-successful traces from 30.5% to 12.8% and introducing no statistically significant increase in unnecessary interventions on aligned traces. These results demonstrate that structured, provenance-based reasoning provides an effective and practical foundation for safeguarding LLM agents from misalignment.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.01236 [cs.CL] |
| (or arXiv:2607.01236v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.01236 arXiv-issued DOI via DataCite |
Submission history
From: Yining She [view email]
[v1]
Fri, 1 May 2026 03:16:35 UTC (468 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.