Learning Safe Agent Behaviour from Human Preferences and Justifications via World Models
Quick Answer
The study introduces DROPJ, a human-centered approach for safely training agent policies in unknown environments using learned world models and human preferences.
Quick Take
The study introduces DROPJ, a human-centered approach for safely training agent policies in unknown environments using learned world models and human preferences. By generating informative simulated trajectories, the method reduces training costs and enhances deployment performance, demonstrating that preference-based feedback significantly improves safety outcomes.
Key Points
- DROPJ uses human preferences to train agent policies in safety-critical environments.
- The method reduces computational costs during training compared to traditional strategies.
- Preference-based feedback significantly improves deployment performance of agents.
- Safety justifications enhance user-defined safety aspects during deployment.
- Real-user experiments validate the effectiveness of the proposed approach.
Paper Resources
📖 Reader Mode
~2 min readAbstract:We address the problem of safely training an agent policy and deploying a good and safe policy, in settings where the environment dynamics are unknown and no suitable reward function is available. In the context of safety-critical environments, we consider traditional reinforcement learning impractical and resort to the resource of human input. We introduce DROPJ, a human-centred method for both safe training and deployment. We first learn a world model (a learned simulator) from a dataset of prior real-world trajectories. A human then plays the game in this learned simulator to extract several informative simulated trajectories. From these, we sample pairs of simulated trajectory segments and elicit from a human their preference over these segments, as well as a reason (justification) for their choice. We then train a reward model from these justified preferences and use it, together with the world model, to directly deploy the agent using model predictive control. Running real-user experiments, we find that generating informative simulated trajectories from a user significantly reduces the computational cost during training compared to other strategies, and can also improve the performance during deployment. In the context of training within a learned simulator, we show that the use of preferences rather than other types of feedback substantially improves the performance during deployment. We further demonstrate that safety justifications accompanying preferences can significantly enhance safety or prioritise user-prescribed aspects of safety associated with them during deployment.
| Comments: | 42 pages, 18 figures. Extended version of a paper presented at ICAART 2026; submitted for consideration in the ICAART 2026 post-publication selected-papers volume in Lecture Notes in Artificial Intelligence |
| Subjects: | Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2607.13172 [cs.AI] |
| (or arXiv:2607.13172v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.13172 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Ilias Kazantzidis [view email]
[v1]
Tue, 14 Jul 2026 18:22:14 UTC (4,713 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →Automatic Ordinary Differential Equations Discovery For Biological Systems Using Large Language Model Powered Agentic System
The MEDA system utilizes large language models and symbolic regression to autonomously discover ordinary differential equations for biological systems, achieving strong structural recovery and biologically plausible models. It outperforms existing methods by integrating domain knowledge and mechanistic constraints, demonstrating effective retrieval and extrapolation capabilities.