COMPASS: Cognitive MCTS-Guided Process Alignment for Safe Search Agents
Quick Take
COMPASS is a Cognitive MCTS-Guided Process Alignment framework that enhances safety in LLM-powered search agents by effectively managing retrieval-induced safety degradation. It utilizes cognitive tree exploration and introspective step-wise alignment to ensure robust safety while maintaining utility, achieving a favorable safety-utility trade-off with significantly less training data.
Key Points
- COMPASS addresses safety degradation in multi-step reasoning for LLM-powered search agents.
- It integrates cognitive tree exploration for efficient synthesis of stealthy attack trajectories.
- Introspective step-wise alignment isolates risky actions for detailed supervision.
- Empirical results show improved safety-utility trade-off with less training data required.
Article Excerpt
From source RSS / original summaryarXiv:2605. 30838v1 Announce Type: new Abstract: LLM-powered search agents enable multi-step reasoning and tool use. However, these capabilities introduce retrieval-induced safety degradation, as harmful intents may decompose into seemingly innocuous sub-queries that lead to unsafe outcomes. Existing alignment methods struggle to capture sparse safety signals and fail to supervise diverse violations across multi-step interactions.
We propose COMPASS, a Cognitive MCTS-Guided Process Alignment framework designed to achieve robust safety alignment throughout the agent workflow while preserving general utility. COMPASS integrates cognitive tree exploration (CTE) to efficiently synthesize stealthy attack trajectories, and introspective step-wise alignment (ISA) to isolate risky intermediate actions for fine-grained process supervision.
Empirical results show that COMPASS achieves a favorable safety-utility trade-off while requiring substantially less training data.
Reader Mode unavailable (could not extract clean content).
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →The Importance of Out-of-Band Metadata for Safe Autonomous Agents: The Redpanda Agentic Data Plane
The Redpanda Agentic Data Plane (ADP) introduces out-of-band metadata channels to enhance the safety of autonomous AI agents, ensuring secure data access and tamper-proof audit trails. This architecture mitigates risks associated with unpredictable AI behavior by enforcing governance throughout the agent lifecycle, demonstrated in a multi-agent trading system with strict data scoping and approval thresholds.