Emergent Communication Fine-Tuning (EC-FT) for Pretrained Language Models
Abstract
It has recently been argued that the currently dominant paradigm in NLP of pretraining on text-only corpora will not yield robust natural language understanding systems. One strain of this argumentation highlights the need for grounded, goal-oriented, and interactive language learning. In this position paper, we articulate how Emergent Communication (EC) can be used in conjunction with large pretrained language models as a `Fine-Tuning' (FT) step (hence, EC-FT) in order to provide them with supervision from such learning scenarios. We discuss methodological issues and difficulties with making this work, and then illustrate the overall idea with a case study in unsupervised machine translation, before concluding with a discussion on the relation to multimodal pretraining.
Cite
Text
Steinert-Threlkeld et al. "Emergent Communication Fine-Tuning (EC-FT) for Pretrained Language Models." ICLR 2022 Workshops: EmeCom, 2022.Markdown
[Steinert-Threlkeld et al. "Emergent Communication Fine-Tuning (EC-FT) for Pretrained Language Models." ICLR 2022 Workshops: EmeCom, 2022.](https://mlanthology.org/iclrw/2022/steinertthrelkeld2022iclrw-emergent/)BibTeX
@inproceedings{steinertthrelkeld2022iclrw-emergent,
title = {{Emergent Communication Fine-Tuning (EC-FT) for Pretrained Language Models}},
author = {Steinert-Threlkeld, Shane and Zhou, Xuhui and Liu, Zeyu and Downey, C. M.},
booktitle = {ICLR 2022 Workshops: EmeCom},
year = {2022},
url = {https://mlanthology.org/iclrw/2022/steinertthrelkeld2022iclrw-emergent/}
}