Fine-Tuning Pre-Trained Language Models Effectively by Optimizing Subnetworks Adaptively
Abstract
Large-scale pre-trained language models have achieved impressive results on a wide range of downstream tasks recently. However, fine-tuning an extremely large-scale pre-trained language model on limited target datasets is often plagued by overfitting and representation degradation. In this paper, we propose a Dynamic Parameter Selection (DPS) algorithm for the large-scale pre-trained models during fine-tuning, which adaptively selects a more promising subnetwork to perform staging updates based on gradients of back-propagation. Experiments on the GLUE benchmark show that DPS outperforms previous fine-tuning methods in terms of overall performance and stability, and consistently achieves better results with variable pre-trained language models. In addition, DPS brings a large magnitude of improvement in out-of-domain transferring experiments and low-resource scenarios, which shows that it can maintain stable general contextual features and reduce the representation collapse. We release our code at \url{https://github.com/ZhangHaojie077/DPS}.
Cite
Text
Zhang et al. "Fine-Tuning Pre-Trained Language Models Effectively by Optimizing Subnetworks Adaptively." Neural Information Processing Systems, 2022.Markdown
[Zhang et al. "Fine-Tuning Pre-Trained Language Models Effectively by Optimizing Subnetworks Adaptively." Neural Information Processing Systems, 2022.](https://mlanthology.org/neurips/2022/zhang2022neurips-finetuning/)BibTeX
@inproceedings{zhang2022neurips-finetuning,
title = {{Fine-Tuning Pre-Trained Language Models Effectively by Optimizing Subnetworks Adaptively}},
author = {Zhang, Haojie and Li, Ge and Li, Jia and Zhang, Zhongjin and Zhu, Yuqi and Jin, Zhi},
booktitle = {Neural Information Processing Systems},
year = {2022},
url = {https://mlanthology.org/neurips/2022/zhang2022neurips-finetuning/}
}