Drop Redundant, Shrink Irrelevant: Selective Knowledge Injection for Language Pretraining
Abstract
Previous research has demonstrated the power of leveraging prior knowledge to improve the performance of deep models in natural language processing. However, traditional methods neglect the fact that redundant and irrelevant knowledge exists in external knowledge bases. In this study, we launched an in-depth empirical investigation into downstream tasks and found that knowledge-enhanced approaches do not always exhibit satisfactory improvements. To this end, we investigate the fundamental reasons for ineffective knowledge infusion and present selective injection for language pretraining, which constitutes a model-agnostic method and is readily pluggable into previous approaches. Experimental results on benchmark datasets demonstrate that our approach can enhance state-of-the-art knowledge injection methods.
Cite
Text
Zhang et al. "Drop Redundant, Shrink Irrelevant: Selective Knowledge Injection for Language Pretraining." International Joint Conference on Artificial Intelligence, 2021. doi:10.24963/IJCAI.2021/552Markdown
[Zhang et al. "Drop Redundant, Shrink Irrelevant: Selective Knowledge Injection for Language Pretraining." International Joint Conference on Artificial Intelligence, 2021.](https://mlanthology.org/ijcai/2021/zhang2021ijcai-drop/) doi:10.24963/IJCAI.2021/552BibTeX
@inproceedings{zhang2021ijcai-drop,
title = {{Drop Redundant, Shrink Irrelevant: Selective Knowledge Injection for Language Pretraining}},
author = {Zhang, Ningyu and Deng, Shumin and Cheng, Xu and Chen, Xi and Zhang, Yichi and Zhang, Wei and Chen, Huajun},
booktitle = {International Joint Conference on Artificial Intelligence},
year = {2021},
pages = {4007-4014},
doi = {10.24963/IJCAI.2021/552},
url = {https://mlanthology.org/ijcai/2021/zhang2021ijcai-drop/}
}