Unsupervised Sentence Embedding Using Document Structure-Based Context
Abstract
We present a new unsupervised method for learning general-purpose sentence embeddings. Unlike existing methods which rely on local contexts, such as words inside the sentence or immediately neighboring sentences, our method selects, for each target sentence, influential sentences from the entire document based on the document structure. We identify a dependency structure of sentences using metadata and text styles. Additionally, we propose an out-of-vocabulary word handling technique for the neural network outputs to model many domain-specific terms which were mostly discarded by existing sentence embedding training methods. We empirically show that the model relies on the proposed dependencies more than the sequential dependency in many cases. We also validate our model on several NLP tasks showing 23% F1-score improvement in coreference resolution in a technical domain and 5% accuracy increase in paraphrase detection compared to baselines.
Cite
Text
Lee and Park. "Unsupervised Sentence Embedding Using Document Structure-Based Context." European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, 2019. doi:10.1007/978-3-030-46147-8_38Markdown
[Lee and Park. "Unsupervised Sentence Embedding Using Document Structure-Based Context." European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, 2019.](https://mlanthology.org/ecmlpkdd/2019/lee2019ecmlpkdd-unsupervised/) doi:10.1007/978-3-030-46147-8_38BibTeX
@inproceedings{lee2019ecmlpkdd-unsupervised,
title = {{Unsupervised Sentence Embedding Using Document Structure-Based Context}},
author = {Lee, Taesung and Park, Youngja},
booktitle = {European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases},
year = {2019},
pages = {633-647},
doi = {10.1007/978-3-030-46147-8_38},
url = {https://mlanthology.org/ecmlpkdd/2019/lee2019ecmlpkdd-unsupervised/}
}