Towards Equipping Transformer with the Ability of Systematic Compositionality
Abstract
One of the key factors in language productivity and human cognition is the ability of Systematic Compositionality, which refers to understanding composed, unseen examples of seen primitives. However, recent evidence reveals that the Transformers have difficulty in generalizing the composed context based on the seen primitives. To this end, we take the first step to propose a compositionality-aware Transformer called CAT and two novel pre-training tasks to facilitate the systematic compositionality. We tentatively provide a successful implementation of a multi-layer CAT on the basis of the especially popular BERT. The experimental results demonstrate that CAT outperforms baselines on compositionality-aware tasks with minimal impact on effectiveness on standardized language understanding tasks.
Cite
Text
Huang et al. "Towards Equipping Transformer with the Ability of Systematic Compositionality." AAAI Conference on Artificial Intelligence, 2024. doi:10.1609/AAAI.V38I16.29788Markdown
[Huang et al. "Towards Equipping Transformer with the Ability of Systematic Compositionality." AAAI Conference on Artificial Intelligence, 2024.](https://mlanthology.org/aaai/2024/huang2024aaai-equipping/) doi:10.1609/AAAI.V38I16.29788BibTeX
@inproceedings{huang2024aaai-equipping,
title = {{Towards Equipping Transformer with the Ability of Systematic Compositionality}},
author = {Huang, Chen and Qin, Peixin and Lei, Wenqiang and Lv, Jiancheng},
booktitle = {AAAI Conference on Artificial Intelligence},
year = {2024},
pages = {18289-18297},
doi = {10.1609/AAAI.V38I16.29788},
url = {https://mlanthology.org/aaai/2024/huang2024aaai-equipping/}
}