Table Pre-Training: A Survey on Model Architectures, Pre-Training Objectives, and Downstream Tasks
Abstract
Following the success of pre-training techniques in the natural language domain, a flurry of table pre-training frameworks have been proposed and have achieved new state-of-the-arts on various downstream tasks such as table question answering, table type recognition, column relation classification, table search, and formula prediction. Various model architectures have been explored to best capture the characteristics of (semi-)structured tables, especially specially-designed attention mechanisms. Moreover, to fully leverage the supervision signals in unlabeled tables, diverse pre-training objectives have been designed and evaluated, for example, denoising cell values, predicting numerical relationships, and learning a neural SQL executor. This survey aims to provide a comprehensive review of model designs, pre-training objectives, and downstream tasks for table pre-training, and we further share our thoughts on existing challenges and future opportunities.
Cite
Text
Dong et al. "Table Pre-Training: A Survey on Model Architectures, Pre-Training Objectives, and Downstream Tasks." International Joint Conference on Artificial Intelligence, 2022. doi:10.24963/IJCAI.2022/761Markdown
[Dong et al. "Table Pre-Training: A Survey on Model Architectures, Pre-Training Objectives, and Downstream Tasks." International Joint Conference on Artificial Intelligence, 2022.](https://mlanthology.org/ijcai/2022/dong2022ijcai-table/) doi:10.24963/IJCAI.2022/761BibTeX
@inproceedings{dong2022ijcai-table,
title = {{Table Pre-Training: A Survey on Model Architectures, Pre-Training Objectives, and Downstream Tasks}},
author = {Dong, Haoyu and Cheng, Zhoujun and He, Xinyi and Zhou, Mengyu and Zhou, Anda and Zhou, Fan and Liu, Ao and Han, Shi and Zhang, Dongmei},
booktitle = {International Joint Conference on Artificial Intelligence},
year = {2022},
pages = {5426-5435},
doi = {10.24963/IJCAI.2022/761},
url = {https://mlanthology.org/ijcai/2022/dong2022ijcai-table/}
}