Generalizing Reward Modeling for Out-of-Distribution Preference Learning

Abstract

Preference learning (PL) with large language models (LLMs) aims to align the LLMs’ generations with human preferences. Previous work on reinforcement learning from human feedback (RLHF) has demonstrated promising results in in-distribution PL. However, due to the difficulty of obtaining human feedback, discretely training reward models (RMs) for every encountered distribution is challenging. Thus, out-of-distribution (OOD) PL is useful for enhancing LLMs’ generalization ability with limited preference feedback. This work addresses OOD PL by optimizing a general RM through a meta-learning approach. A bilevel optimization algorithm is utilized during meta-training to learn an RM that guides policy learning to align with human preferences across various distributions. When encountering a test distribution, the meta-test procedure optimizes regularized policy using the learned RM for PL. We theoretically demonstrate the convergence rate of the bilevel optimization algorithm under reasonable assumptions. Additionally, we conduct experiments on two text generation tasks across 22 held-out data distributions and outperform various strong baselines across various evaluation metrics.