When Aggregate Alignment Misleads: Auditing Policy Repair Without Per-State Expert Actions
Quick Answer
This paper shows that A multi-restart LLM editor achieves a RevPAR of 108.47, closely matching the benchmark of 108.75, despite limited diagnostic feedback in a hotel-pricing simulator.
Quick Take
A multi-restart LLM editor achieves a RevPAR of 108.47, closely matching the benchmark of 108.75, despite limited diagnostic feedback in a hotel-pricing simulator. This study highlights the importance of evaluating agentic policy repairs through reliable closed-loop outcomes rather than solely behavioral distance.
Key Points
- The LLM editor's performance gap from the benchmark is -0.276 with a 95% CI of [-0.692, 0.146].
- A cheap diagnostic projection recovers much revenue, achieving 107.90 RevPAR.
- Non-semantic proposers fall short by 8.77 - 14.57 RevPAR points compared to the LLM editor.
- A tree editor shows stronger alignment but results in lower revenue at 98.91 RevPAR.
- The study suggests evaluating policy repairs based on reliable outcomes rather than single behavioral distances.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Agentic AI systems are increasingly used to edit, refine, and repair decision policies, but evaluating these edits is difficult when per-state expert action labels are unavailable. We study this problem in a hotel-pricing simulator where an agentic policy editor receives only region-level diagnostic feedback: summaries of how its price distribution differs from a benchmark policy across time, inventory, and market regions. The editor cannot observe benchmark actions, benchmark source code, reward numbers, or held-out outcomes, and may only propose constrained edits to a target-action table. On 5,000 held-out episodes, a multi-restart LLM editor reaches RevPAR 108.47 (95% CI 107.61 - 109.34), close to the benchmark policy's 108.75 (107.81 - 109.68), with paired gap (LLM minus benchmark) -0.276 and 95% CI [-0.692, 0.146]. A cheap diagnostic projection already recovers much of the revenue (107.90), so the LLM editor's distinctive gain is not raw revenue lift alone: it also reduces episode composition distance from 1.153 to 0.609. This is the strongest non-benchmark repair result. This profile is not explained by restart search alone: non-semantic proposers with up to 2,500 evaluations fall 8.77 - 14.57 RevPAR points short. Nor is it explained by plausible prompt format: a shuffled-diagnostic control breaks region-error correspondence and falls to RevPAR 94.30. The match is genuine but partial. A tree editor achieves stronger pooled alignment, 0.214 versus 0.266, and stronger reference-state D1, 0.328 versus 1.197, yet revenue falls to 98.91. These results show that agentic policy repair should be evaluated by whether diagnostic feedback becomes reliable closed-loop outcome, not by a single behavioral distance.
| Subjects: | Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2607.03386 [cs.AI] |
| (or arXiv:2607.03386v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.03386 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Sidi Chang [view email]
[v1]
Fri, 3 Jul 2026 14:42:01 UTC (18 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →Procedural Memory Distillation: Online Reflection for Self-Improving Language Models
Procedural Memory Distillation (PMD) enhances reinforcement learning by converting cross-episode signals into reusable memory, improving Qwen3-8B and OLMo3-Instruct-7B models by 3.8-5.5% on SCIKNOWEVAL and 7.9-13.6% on . The co-evolution of policy and memory allows for more effective self-supervision, demonstrating significant performance gains when both components are active.