Repurposing Decoder-Transformer Language Models for Abstractive Summarization
Abstract
Neural network models have shown excellent fluency and performance when applied to abstractive summarization. Many approaches to neural abstractive summarization involve the introduction of significant inductive bias, such as pointer-generator architectures, coverage, and partially extractive procedures, designed to mimic human summarization. We show that it is possible to attain competitive performance by instead directly viewing summarization as language modeling. We introduce a simple procedure built upon pre-trained decoder-transformers to obtain competitive ROUGE scores using a language modeling loss alone, with no beam-search or other decoding-time optimization, and instead rely on efficient nucleus sampling and greedy decoding.
Cite
Text
de Oliveira and Rodrigo. "Repurposing Decoder-Transformer Language Models for Abstractive Summarization." NeurIPS 2019 Workshops: Document_Intelligence, 2019.Markdown
[de Oliveira and Rodrigo. "Repurposing Decoder-Transformer Language Models for Abstractive Summarization." NeurIPS 2019 Workshops: Document_Intelligence, 2019.](https://mlanthology.org/neuripsw/2019/deoliveira2019neuripsw-repurposing/)BibTeX
@inproceedings{deoliveira2019neuripsw-repurposing,
title = {{Repurposing Decoder-Transformer Language Models for Abstractive Summarization}},
author = {de Oliveira, Luke and Rodrigo, Alfredo Láinez},
booktitle = {NeurIPS 2019 Workshops: Document_Intelligence},
year = {2019},
url = {https://mlanthology.org/neuripsw/2019/deoliveira2019neuripsw-repurposing/}
}