POLAR-Bench: A Diagnostic Benchmark for Privacy-Utility Trade-offs in LLM Agents
Quick Answer
POLAR-Bench introduces a diagnostic benchmark assessing privacy-utility trade-offs in LLM agents, revealing that current frontier models retain over 99% of protected attributes, while smaller models leak more than 50%.
Quick Take
This benchmark evaluates 7,852 samples across 10 domains, highlighting the need for improved privacy alignment in user-trusted agents.
Key Points
- POLAR-Bench scores privacy and utility across 10 domains with 7,852 samples.
- Current frontier models withhold over 99% of protected attributes.
- Smaller open-weight models (1-30B) score significantly worse, with some leaking over 50%.
- Benchmark helps identify breakdowns in intent-following for privacy alignment.
- Adversarial probing assesses both task-relevant and protected attributes.
Paper Resources
📖 Reader Mode
~2 min readAbstract:LLM agents increasingly have access to private user data and act on the user's behalf when interacting with third-party systems. The user defines what may and must not be shared, and the agent must robustly follow that intent even when third-party systems behave adversarially. We introduce POLAR-Bench (Policy-aware adversarial Benchmark), in which a trusted model with a privacy policy and a task converses with a third-party model that adversarially probes for both task-relevant and protected attributes. Across 10 domains and 7,852 samples, we score privacy and utility by deterministic set-membership and vary privacy policy dimension and attack strategy along two orthogonal axes, producing a 5 times 5 diagnostic surface per model. Our results reveal a sharp split: current frontier models withhold over 99% of protected attributes, while smaller open-weight models in the 1--30B range, the class users most commonly run as their own trusted agent on-device or via private inference, score notably worse, with the weakest leaking over half. POLAR-Bench thus localizes where each model's intent-following breaks down, providing a foothold for privacy alignment where it matters most.
| Comments: | Preprint |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2605.19127 [cs.AI] |
| (or arXiv:2605.19127v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2605.19127 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Qiaoyuan Zheng [view email]
[v1]
Mon, 18 May 2026 21:27:07 UTC (4,346 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
HOBA (Hierarchical On-policy Bidding Agents) is a novel hierarchical reinforcement learning framework that enhances online advertising bidding systems by improving adaptability and reducing hyperparameter tuning costs. It utilizes a for hyperparameter inference, a SARSA agent for expert model selection, and a dynamic expert pool for bid execution, achieving a +3.6% increase in target cost during large-scale deployment and outperforming state-of-the-art baselines on AuctionNet.