Conditional Language Policy: A General Framework for Steerable Multi-Objective Finetuning
Abstract
Reward-based finetuning is crucial for aligning language policies with intended behaviors (e.g., creativity and safety). A key challenge is to develop steerable language models that trade-off multiple (conflicting) objectives in a flexible and efficient manner. This paper presents Conditional Language Policy (CLP), a general framework for finetuning language models on multiple objectives. Building on techniques from multi-task training and parameter-efficient finetuning, CLP learn steerable models that effectively trade-off conflicting objectives at inference time. Notably, this does not require training or maintaining multiple models to achieve different trade-offs between the objectives. Through extensive experiments and ablations on two summarization datasets, we show that CLP learns steerable language models that outperform and Pareto-dominate the existing approaches for multi-objective finetuning.
Cite
Text
Wang et al. "Conditional Language Policy: A General Framework for Steerable Multi-Objective Finetuning." NeurIPS 2024 Workshops: Pluralistic-Alignment, 2024.Markdown
[Wang et al. "Conditional Language Policy: A General Framework for Steerable Multi-Objective Finetuning." NeurIPS 2024 Workshops: Pluralistic-Alignment, 2024.](https://mlanthology.org/neuripsw/2024/wang2024neuripsw-conditional/)BibTeX
@inproceedings{wang2024neuripsw-conditional,
title = {{Conditional Language Policy: A General Framework for Steerable Multi-Objective Finetuning}},
author = {Wang, Kaiwen and Kidambi, Rahul and Sullivan, Ryan and Agarwal, Alekh and Dann, Christoph and Michi, Andrea and Gelmi, Marco and Li, Yunxuan and Gupta, Raghav and Dubey, Kumar Avinava and Rame, Alexandre and Ferret, Johan and Cideron, Geoffrey and Hou, Le and Yu, Hongkun and Ahmed, Amr and Mehta, Aranyak and Hussenot, Leonard and Bachem, Olivier and Leurent, Edouard},
booktitle = {NeurIPS 2024 Workshops: Pluralistic-Alignment},
year = {2024},
url = {https://mlanthology.org/neuripsw/2024/wang2024neuripsw-conditional/}
}