Learning Fair Policies in Multi-Objective Preference-Based Reinforcement Learning

Abstract

In this paper, we address the issue of fairness in preference-based reinforcement learning (PbRL) in the presence of multiple objectives. The main objective is to design control policies that can optimize multiple objectives while treating each objective fairly. Toward this objective, we design a new fairness-induced preference-based reinforcement learning or FPbRL. The main idea of FPbRL is to learn vector reward functions associated with multiple objectives via new welfare-based preferences rather than reward-based preferences in PbRL, coupled with policy learning via maximizing a welfare function. Finally, we conduct experiments to compare FPbRL and other relevant techniques in different real-world environments, considering synthetic human teacher preferences with various imperfection and irrationality factors, including myopic behavior, deviations, and perturbed feedback. Our study shows that the proposed FPbRL approach strikes a balance between efficiency and equity, effectively learning policies that are both efficient and impartial even in the presence of imperfect preferences.