Attack Selection in Agentic AI Control Evaluations Meaningfully Decreases Safety
Quick Answer
This paper shows that Strategic attack selection in AI control evaluations significantly reduces safety, with a 20pp drop in BashArena and LinuxArena at a 1% audit budget.
Quick Take
Current evaluations may underestimate risks from selective attackers, necessitating revised methodologies for realistic safety estimates.
Key Points
- Attack selection reduces empirical safety by 20pp in both BashArena and LinuxArena.
- Start and stop policies for attackers significantly impact safety evaluations.
- Existing evaluations may yield overly optimistic safety estimates against selective attackers.
- A 1% audit budget shows substantial safety reductions with strategic attack policies.
- Future evaluations should incorporate attack selection for accurate safety assessments.
Paper Resources
Article Content
From source RSS / original summaryarXiv:2606. 06529v1 Announce Type: new Abstract: An attacker that strategically chooses when to attack is much harder to catch than one that attacks indiscriminately. AI control is a safety framework for deploying capable but untrusted AI agents under the oversight of a weaker, trusted monitor and a limited human audit budget.
Control evaluations stress-test these protocols by pitting a red-team attack policy against the blue-team monitor, but current evaluations typically assume attackers that do not strategically select when to attack. We study this capability, attack selection, in agentic settings by decomposing attack decisions into a start policy, which decides when an attacker should attack, and a stop policy, which decides when an attacker should abort an ongoing attack.
Across two agentic settings, BashArena and LinuxArena, both policies substantially lower measured empirical safety without changing the underlying attack capability. At a 1% audit budget, our start policy reduces safety by 20pp on both BashArena and LinuxArena, and our stop policy reduces safety by 20pp on BashArena and 28pp on LinuxArena. These reductions should be interpreted as upper bounds on the effect of attack selection.
Existing control evaluations may therefore yield overly optimistic safety estimates against selective attackers. We recommend that future evaluations, system cards, and safety cases elicit attack selection to produce more realistic safety estimates.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
HOBA (Hierarchical On-policy Bidding Agents) is a novel hierarchical reinforcement learning framework that enhances online advertising bidding systems by improving adaptability and reducing hyperparameter tuning costs. It utilizes a for hyperparameter inference, a SARSA agent for expert model selection, and a dynamic expert pool for bid execution, achieving a +3.6% increase in target cost during large-scale deployment and outperforming state-of-the-art baselines on AuctionNet.