Learning Safe Agent Behaviour from Human Preferences and Justifications via World Models
Quick Answer
The study introduces DROPJ, a human-centered approach for safely training agent policies in unknown environments using learned world models and human preferences.
Quick Take
By generating informative simulated trajectories, the method reduces training costs and enhances deployment performance, demonstrating that preference-based feedback significantly improves safety outcomes.
Key Points
- DROPJ uses human preferences to train agent policies in safety-critical environments.
- The method reduces computational costs during training compared to traditional strategies.
- Preference-based feedback significantly improves deployment performance of agents.
- Safety justifications enhance user-defined safety aspects during deployment.
- Real-user experiments validate the effectiveness of the proposed approach.
Paper Resources
📖 Reader Mode
~2 min readAbstract:We address the problem of safely training an agent policy and deploying a good and safe policy, in settings where the environment dynamics are unknown and no suitable reward function is available. In the context of safety-critical environments, we consider traditional reinforcement learning impractical and resort to the resource of human input. We introduce DROPJ, a human-centred method for both safe training and deployment. We first learn a world model (a learned simulator) from a dataset of prior real-world trajectories. A human then plays the game in this learned simulator to extract several informative simulated trajectories. From these, we sample pairs of simulated trajectory segments and elicit from a human their preference over these segments, as well as a reason (justification) for their choice. We then train a reward model from these justified preferences and use it, together with the world model, to directly deploy the agent using model predictive control. Running real-user experiments, we find that generating informative simulated trajectories from a user significantly reduces the computational cost during training compared to other strategies, and can also improve the performance during deployment. In the context of training within a learned simulator, we show that the use of preferences rather than other types of feedback substantially improves the performance during deployment. We further demonstrate that safety justifications accompanying preferences can significantly enhance safety or prioritise user-prescribed aspects of safety associated with them during deployment.
| Comments: | 42 pages, 18 figures. Extended version of a paper presented at ICAART 2026; submitted for consideration in the ICAART 2026 post-publication selected-papers volume in Lecture Notes in Artificial Intelligence |
| Subjects: | Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2607.13172 [cs.AI] |
| (or arXiv:2607.13172v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.13172 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Ilias Kazantzidis [view email]
[v1]
Tue, 14 Jul 2026 18:22:14 UTC (4,713 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
HOBA (Hierarchical On-policy Bidding Agents) is a novel hierarchical reinforcement learning framework that enhances online advertising bidding systems by improving adaptability and reducing hyperparameter tuning costs. It utilizes a for hyperparameter inference, a SARSA agent for expert model selection, and a dynamic expert pool for bid execution, achieving a +3.6% increase in target cost during large-scale deployment and outperforming state-of-the-art baselines on AuctionNet.