Beyond Goodhart's Law: A Dynamic Benchmark for Evaluating Compliance in Multi-Agent Systems
Quick Answer
This paper shows that The introduction of MAC-Bench addresses compliance issues in multi-agent systems, revealing trade-offs between task success and regulatory adherence.
Quick Take
Using the SERV pipeline, it transforms legal texts into executable scenarios, highlighting the Compliance-Weighted Success Rate and Machiavellian Gap metrics. This benchmark exposes the risks of 'Machiavellian' behaviors in autonomous agents, crucial for evaluating .
Key Points
- MAC-Bench evaluates procedural alignment under realistic pressures in .
- The SERV pipeline converts unstructured legal texts into executable scenarios.
- New metrics include Compliance-Weighted Success Rate (CSR) and Machiavellian Gap (MG).
- The benchmark reveals significant trade-offs between task success and compliance.
- Current frameworks often overlook procedural compliance, leading to risky agent behaviors.
Paper Resources
Article Content
From source RSS / original summaryarXiv:2606. 07805v1 Announce Type: new Abstract: The rapid evolution of (LLMs) from passive assistants to autonomous, execution-capable agents has introduced critical operational risks. Most current evaluation frameworks neglect procedural compliance, leading to ''Machiavellian'' behaviors where agents strategically violate safety rules to maximize rewards - a direct manifestation of Goodhart's Law.
To address this blind spot, we introduce MAC-Bench, a dynamic, adversarial benchmark designed to evaluate the procedural alignment of under realistic pressure. We propose the SERV(Seed - Evolve - Refine - Verify) pipeline, an ``Agent-as-a-Benchmark'' paradigm that transforms unstructured legal texts into executable, contamination-free scenarios.
By synthesizing holographic sandbox environments and injecting calibrated social-engineering pressure vectors, MAC-Bench forces agents into Pareto-optimal trade-offs between task success and regulatory adherence. We introduced novel metrics: the Compliance-Weighted Success Rate (CSR) and the Machiavellian Gap (MG), and conducted a comprehensive evaluation of state-of-the-art frontier models to reveal the pervasive trade-offs between success and compliance.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
HOBA (Hierarchical On-policy Bidding Agents) is a novel hierarchical reinforcement learning framework that enhances online advertising bidding systems by improving adaptability and reducing hyperparameter tuning costs. It utilizes a for hyperparameter inference, a SARSA agent for expert model selection, and a dynamic expert pool for bid execution, achieving a +3.6% increase in target cost during large-scale deployment and outperforming state-of-the-art baselines on AuctionNet.