From Cloze to Comprehension: Retrofitting Pre-Trained Masked Language Models to Pre-Trained Machine Reader

Weiwen Xu, Xin Li, Wenxuan Zhang, Meng Zhou, Wai Lam, Luo Si, Lidong Bing

NeurIPS 2023

/neurips/2023/xu2023neurips-cloze/

Abstract

We present Pre-trained Machine Reader (PMR), a novel method for retrofitting pre-trained masked language models (MLMs) to pre-trained machine reading comprehension (MRC) models without acquiring labeled data.PMR can resolve the discrepancy between model pre-training and downstream fine-tuning of existing MLMs.To build the proposed PMR, we constructed a large volume of general-purpose and high-quality MRC-style training data by using Wikipedia hyperlinks and designed a Wiki Anchor Extraction task to guide the MRC-style pre-training.Apart from its simplicity, PMR effectively solves extraction tasks, such as Extractive Question Answering and Named Entity Recognition. PMR shows tremendous improvements over existing approaches, especially in low-resource scenarios.When applied to the sequence classification task in the MRC formulation, PMR enables the extraction of high-quality rationales to explain the classification process, thereby providing greater prediction explainability. PMR also has the potential to serve as a unified model for tackling various extraction and classification tasks in the MRC formulation.

PDF NeurIPS OpenReview Semantic Scholar

Cite

Text

Xu et al. "From Cloze to Comprehension: Retrofitting Pre-Trained Masked Language Models to Pre-Trained Machine Reader." Neural Information Processing Systems, 2023.

Markdown

[Xu et al. "From Cloze to Comprehension: Retrofitting Pre-Trained Masked Language Models to Pre-Trained Machine Reader." Neural Information Processing Systems, 2023.](https://mlanthology.org/neurips/2023/xu2023neurips-cloze/)

BibTeX

@inproceedings{xu2023neurips-cloze,
  title     = {{From Cloze to Comprehension: Retrofitting Pre-Trained Masked Language Models to Pre-Trained Machine Reader}},
  author    = {Xu, Weiwen and Li, Xin and Zhang, Wenxuan and Zhou, Meng and Lam, Wai and Si, Luo and Bing, Lidong},
  booktitle = {Neural Information Processing Systems},
  year      = {2023},
  url       = {https://mlanthology.org/neurips/2023/xu2023neurips-cloze/}
}