Weighted Training for Cross-Task Learning
Abstract
In this paper, we introduce Target-Aware Weighted Training (TAWT), a weighted training algorithm for cross-task learning based on minimizing a representation-based task distance between the source and target tasks. We show that TAWT is easy to implement, is computationally efficient, requires little hyperparameter tuning, and enjoys non-asymptotic learning-theoretic guarantees. The effectiveness of TAWT is corroborated through extensive experiments with BERT on four sequence tagging tasks in natural language processing (NLP), including part-of-speech (PoS) tagging, chunking, predicate detection, and named entity recognition (NER). As a byproduct, the proposed representation-based task distance allows one to reason in a theoretically principled way about several critical aspects of cross-task learning, such as the choice of the source data and the impact of fine-tuning.
Cite
Text
Chen et al. "Weighted Training for Cross-Task Learning." International Conference on Learning Representations, 2022.Markdown
[Chen et al. "Weighted Training for Cross-Task Learning." International Conference on Learning Representations, 2022.](https://mlanthology.org/iclr/2022/chen2022iclr-weighted/)BibTeX
@inproceedings{chen2022iclr-weighted,
title = {{Weighted Training for Cross-Task Learning}},
author = {Chen, Shuxiao and Crammer, Koby and He, Hangfeng and Roth, Dan and Su, Weijie J},
booktitle = {International Conference on Learning Representations},
year = {2022},
url = {https://mlanthology.org/iclr/2022/chen2022iclr-weighted/}
}