Feedback Manipulation Regularization: Enabling Offline Agent Alignment for Imitation Learning
Quick Answer
The paper introduces Feedback Manipulation Regularization (FMR), an algorithm-agnostic method that enhances imitation learning alignment by utilizing evaluative feedback as a corrective signal.
Quick Take
The paper introduces Feedback Manipulation Regularization (FMR), an algorithm-agnostic method that enhances imitation learning alignment by utilizing evaluative feedback as a corrective signal. It shows up to a 98% reduction in misalignment across various algorithms in Safety Gymnasium environments, proving effective even in data-scarce scenarios.
Key Points
- FMR improves imitation learning policies by leveraging evaluative feedback.
- Demonstrated up to 98% reduction in misalignment across imitation learning algorithms.
- Adapted Safety Gymnasium as a testbed for alignment evaluation.
- Robust performance in limited data regimes with noisy demonstrations.
- Addresses the gap in using interconnected signals for offline training.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Reinforcement learning (RL) research has increasingly shifted focus towards alignment, ensuring agents learn behaviors adhering to human values. While human demonstrations and feedback have proven crucial for alignment, existing approaches predominantly combine these signals using multi-stage pipelines designed for the contextual bandit framing of language generation. Yet little work explores how these complementary inputs can serve as a richer, interconnected signal for single-stage offline training in fully sequential decision-making environments. We propose Feedback Manipulation Regularization (FMR), an algorithm-agnostic method that harnesses evaluative feedback as a corrective signal to improve the alignment of imitation learning policies. We adapt Safety Gymnasium environments to be a principled testbed for alignment evaluation, demonstrating improved aptitude and up to a 98\% reduction in misalignment across a range of imitation learning algorithms. FMR remains robust in limited data regimes, even when learning from scarce aligned and uninformative noisy demonstrations.
| Subjects: | Artificial Intelligence (cs.AI); Human-Computer Interaction (cs.HC); Machine Learning (cs.LG) |
| Cite as: | arXiv:2607.07859 [cs.AI] |
| (or arXiv:2607.07859v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.07859 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Benjamin Poole [view email]
[v1]
Wed, 8 Jul 2026 18:47:29 UTC (7,038 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →Adversarial Social Epistemology for Assemblies of Humans and Large Language Models
The paper introduces Adversarial Social Epistemology (ASE) to analyze how agents manipulate trust in public communications, highlighting mechanisms that undermine the reliability of testimony and inference. It critiques existing frameworks like epistemic bubbles and misinformation diffusion, proposing a new language for understanding trust breaches and auditing inferential chains in densely interactive environments involving humans and large language models.