Feedback Manipulation Regularization: Enabling Offline Agent Alignment for Imitation Learning
Quick Answer
The paper introduces Feedback Manipulation Regularization (FMR), an algorithm-agnostic method that enhances imitation learning alignment by utilizing evaluative feedback as a corrective signal.
Quick Take
It shows up to a 98% reduction in misalignment across various algorithms in Safety Gymnasium environments, proving effective even in data-scarce scenarios.
Key Points
- FMR improves imitation learning policies by leveraging evaluative feedback.
- Demonstrated up to 98% reduction in misalignment across imitation learning algorithms.
- Adapted Safety Gymnasium as a testbed for alignment evaluation.
- Robust performance in limited data regimes with noisy demonstrations.
- Addresses the gap in using interconnected signals for offline training.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Reinforcement learning (RL) research has increasingly shifted focus towards alignment, ensuring agents learn behaviors adhering to human values. While human demonstrations and feedback have proven crucial for alignment, existing approaches predominantly combine these signals using multi-stage pipelines designed for the contextual bandit framing of language generation. Yet little work explores how these complementary inputs can serve as a richer, interconnected signal for single-stage offline training in fully sequential decision-making environments. We propose Feedback Manipulation Regularization (FMR), an algorithm-agnostic method that harnesses evaluative feedback as a corrective signal to improve the alignment of imitation learning policies. We adapt Safety Gymnasium environments to be a principled testbed for alignment evaluation, demonstrating improved aptitude and up to a 98\% reduction in misalignment across a range of imitation learning algorithms. FMR remains robust in limited data regimes, even when learning from scarce aligned and uninformative noisy demonstrations.
| Subjects: | Artificial Intelligence (cs.AI); Human-Computer Interaction (cs.HC); Machine Learning (cs.LG) |
| Cite as: | arXiv:2607.07859 [cs.AI] |
| (or arXiv:2607.07859v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.07859 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Benjamin Poole [view email]
[v1]
Wed, 8 Jul 2026 18:47:29 UTC (7,038 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
HOBA (Hierarchical On-policy Bidding Agents) is a novel hierarchical reinforcement learning framework that enhances online advertising bidding systems by improving adaptability and reducing hyperparameter tuning costs. It utilizes a for hyperparameter inference, a SARSA agent for expert model selection, and a dynamic expert pool for bid execution, achieving a +3.6% increase in target cost during large-scale deployment and outperforming state-of-the-art baselines on AuctionNet.