Privately Aligning Language Models with Reinforcement Learning
Abstract
Positioned between pre-training and user deployment, aligning large language models (LLMs) through reinforcement learning (RL) has emerged as a prevailing strategy for training instruction following-models such as ChatGPT. In this work, we initiate the study of privacy-preserving alignment of LLMs through Differential Privacy (DP) in conjunction with RL. Following the influential work of Ziegler et al. (2020), we study two dominant paradigms: (i) alignment via RL without human in the loop (e.g., positive review generation) and (ii) alignment via RL from human feedback (RLHF) (e.g., summarization in a human-preferred way). We give a new DP framework to achieve alignment via RL, and prove its correctness. Our experimental results validate the effectiveness of our approach, offering competitive utility while ensuring strong privacy protections.
Cite
Text
Wu et al. "Privately Aligning Language Models with Reinforcement Learning." International Conference on Learning Representations, 2024.Markdown
[Wu et al. "Privately Aligning Language Models with Reinforcement Learning." International Conference on Learning Representations, 2024.](https://mlanthology.org/iclr/2024/wu2024iclr-privately/)BibTeX
@inproceedings{wu2024iclr-privately,
title = {{Privately Aligning Language Models with Reinforcement Learning}},
author = {Wu, Fan and Inan, Huseyin A and Backurs, Arturs and Chandrasekaran, Varun and Kulkarni, Janardhan and Sim, Robert},
booktitle = {International Conference on Learning Representations},
year = {2024},
url = {https://mlanthology.org/iclr/2024/wu2024iclr-privately/}
}