Image-to-Markup Generation via Paired Adversarial Learning
Abstract
Motivated by the fact that humans can grasp semantic-invariant features shared by the same category while attention-based models focus mainly on discriminative features of each object, we propose a scalable paired adversarial learning (PAL) method for image-to-markup generation. PAL can incorporate the prior knowledge of standard templates to guide the attention-based model for discovering semantic-invariant features when the model pays attention to regions of interest. Furthermore, we also extend the convolutional attention mechanism to speed up the image-to-markup parsing process while achieving competitive performance compared with recurrent attention models. We evaluate the proposed method in the scenario of handwritten-image-to-LaTeX generation, i.e., converting handwritten mathematical expressions to LaTeX. Experimental results show that our method can significantly improve the generalization performance over standard attention-based encoder-decoder models.
Cite
Text
Wu et al. "Image-to-Markup Generation via Paired Adversarial Learning." European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, 2018. doi:10.1007/978-3-030-10925-7_2Markdown
[Wu et al. "Image-to-Markup Generation via Paired Adversarial Learning." European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, 2018.](https://mlanthology.org/ecmlpkdd/2018/wu2018ecmlpkdd-imagetomarkup/) doi:10.1007/978-3-030-10925-7_2BibTeX
@inproceedings{wu2018ecmlpkdd-imagetomarkup,
title = {{Image-to-Markup Generation via Paired Adversarial Learning}},
author = {Wu, Jin-Wen and Yin, Fei and Zhang, Yan-Ming and Zhang, Xu-Yao and Liu, Cheng-Lin},
booktitle = {European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases},
year = {2018},
pages = {18-34},
doi = {10.1007/978-3-030-10925-7_2},
url = {https://mlanthology.org/ecmlpkdd/2018/wu2018ecmlpkdd-imagetomarkup/}
}