COMPASS: Cognitive MCTS-Guided Process Alignment for Safe Search Agents
Quick Answer
COMPASS is a Cognitive MCTS-Guided Process Alignment framework that enhances safety in LLM-powered search agents by effectively managing retrieval-induced safety degradation.
Quick Take
COMPASS is a Cognitive MCTS-Guided Process Alignment framework that enhances safety in LLM-powered search agents by effectively managing retrieval-induced safety degradation. It utilizes cognitive tree exploration and introspective step-wise alignment to ensure robust safety while maintaining utility, achieving a favorable safety-utility trade-off with significantly less training data.
Key Points
- COMPASS addresses safety degradation in multi-step reasoning for LLM-powered search agents.
- It integrates cognitive tree exploration for efficient synthesis of stealthy attack trajectories.
- Introspective step-wise alignment isolates risky actions for detailed supervision.
- Empirical results show improved safety-utility trade-off with less training data required.
Paper Resources
Article Excerpt
From source RSS / original summaryarXiv:2605. 30838v1 Announce Type: new Abstract: LLM-powered search agents enable multi-step reasoning and . However, these capabilities introduce retrieval-induced safety degradation, as harmful intents may decompose into seemingly innocuous sub-queries that lead to unsafe outcomes. Existing alignment methods struggle to capture sparse safety signals and fail to supervise diverse violations across multi-step interactions.
We propose COMPASS, a Cognitive MCTS-Guided Process Alignment framework designed to achieve robust safety alignment throughout the agent workflow while preserving general utility. COMPASS integrates cognitive tree exploration (CTE) to efficiently synthesize stealthy attack trajectories, and introspective step-wise alignment (ISA) to isolate risky intermediate actions for fine-grained process supervision.
Empirical results show that COMPASS achieves a favorable safety-utility trade-off while requiring substantially less training data.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →Automatic Ordinary Differential Equations Discovery For Biological Systems Using Large Language Model Powered Agentic System
The MEDA system utilizes large language models and symbolic regression to autonomously discover ordinary differential equations for biological systems, achieving strong structural recovery and biologically plausible models. It outperforms existing methods by integrating domain knowledge and mechanistic constraints, demonstrating effective retrieval and extrapolation capabilities.