Conditional Adapters: Parameter-Efficient Transfer Learning with Fast Inference
Abstract
We propose Conditional Adapter (CoDA), a parameter-efficient transfer learning method that also improves inference efficiency. CoDA generalizes beyond standard adapter approaches to enable a new way of balancing speed and accuracy using conditional computation.Starting with an existing dense pretrained model, CoDA adds sparse activation together with a small number of new parameters and a light-weight training phase.Our experiments demonstrate that the CoDA approach provides an unexpectedly efficient way to transfer knowledge.Across a variety of language, vision, and speech tasks, CoDA achieves a 2x to 8x inference speed-up compared to the state-of-the-art Adapter approaches with moderate to no accuracy loss and the same parameter efficiency.
Cite
Text
Lei et al. "Conditional Adapters: Parameter-Efficient Transfer Learning with Fast Inference." Neural Information Processing Systems, 2023.Markdown
[Lei et al. "Conditional Adapters: Parameter-Efficient Transfer Learning with Fast Inference." Neural Information Processing Systems, 2023.](https://mlanthology.org/neurips/2023/lei2023neurips-conditional/)BibTeX
@inproceedings{lei2023neurips-conditional,
title = {{Conditional Adapters: Parameter-Efficient Transfer Learning with Fast Inference}},
author = {Lei, Tao and Bai, Junwen and Brahma, Siddhartha and Ainslie, Joshua and Lee, Kenton and Zhou, Yanqi and Du, Nan and Zhao, Vincent and Wu, Yuexin and Li, Bo and Zhang, Yu and Chang, Ming-Wei},
booktitle = {Neural Information Processing Systems},
year = {2023},
url = {https://mlanthology.org/neurips/2023/lei2023neurips-conditional/}
}