Safe Reward Learning from Human Preferences and Justifications

Abstract

We address the problem of learning autonomous safe agent behaviour with unknown dynamics and reward functions, where traditional Reinforcement Learning is impossible. We present DROPJ, a human-centred algorithm that maximises safety during both training and deployment. We first learn a world model (a learned simulation) from a set of past real-world trajectories. A user then plays the game in the simulation to draw several informative virtual trajectories. From these, we extract pairs of trajectory segments and present them to a user to elicit their preference over these segments and the reason (justification) for that preference. With this feedback, a reward model is trained, which is used to deploy the agent with Model Predictive Control. We find that generating trajectories from user trials significantly reduces the computational cost of training, and significantly improves performance during deployment. In that context, we show that the use of preferences rather than other types of feedback substantially improves the performance. We further demonstrate that the use of justifications associated with safety requirements results in safer policies.

More information

Identifiers

ORCID for Ilias Kazantzidis: orcid.org/0000-0002-1127-3843

ORCID for Timothy Norman: orcid.org/0000-0002-6387-4034

ORCID for Chris Freeman: orcid.org/0000-0003-0305-9246

Catalogue record

Export record

Altmetrics