Composing Knowledge and Compression Interventions for Language Models
Abstract
Test-time interventions for language models aim to enhance factual accuracy, reduce harmful outputs, and improve model efficiency while avoiding excessive training costs. But existing interventions are developing independently. In practice, multiple interventions must be applied to the same model sequentially. We introduce composable interventions, a framework for studying the impacts of repeatedly intervening on the same language model. To showcase our framework, we compose interventions for two burgeoning interventions: knowledge editing and model compression. We find that compression undoes knowledge edits faster than it decays general model performance. We also find that compressing models makes them harder to edit and show that composing interventions impacts predicted logits.
Cite
Text
Kolbeinsson et al. "Composing Knowledge and Compression Interventions for Language Models." ICLR 2024 Workshops: R2-FM, 2024.Markdown
[Kolbeinsson et al. "Composing Knowledge and Compression Interventions for Language Models." ICLR 2024 Workshops: R2-FM, 2024.](https://mlanthology.org/iclrw/2024/kolbeinsson2024iclrw-composing/)BibTeX
@inproceedings{kolbeinsson2024iclrw-composing,
title = {{Composing Knowledge and Compression Interventions for Language Models}},
author = {Kolbeinsson, Arinbjörn and Huang, Tianjin and Gao, Shanghua and Liu, Shiwei and Schwarz, Jonathan Richard and Vaidya, Anurag Jayant and Mahmood, Faisal and Zitnik, Marinka and Chen, Tianlong and Hartvigsen, Thomas},
booktitle = {ICLR 2024 Workshops: R2-FM},
year = {2024},
url = {https://mlanthology.org/iclrw/2024/kolbeinsson2024iclrw-composing/}
}