Sequence Length Independent Norm-Based Generalization Bounds for Transformers
Abstract
This paper provides norm-based generalization bounds for the Transformer architecture that do not depend on the input sequence length. We employ a covering number based approach to prove our bounds. We use three novel covering number bounds for the function class of bounded linear mappings to upper bound the Rademacher complexity of the Transformer. Furthermore, we show this generalization bound applies to the common Transformer training technique of masking and then predicting the masked word. We also run a simulated study on a sparse majority data set that empirically validates our theoretical findings.
Cite
Text
Trauger and Tewari. "Sequence Length Independent Norm-Based Generalization Bounds for Transformers." Artificial Intelligence and Statistics, 2024.Markdown
[Trauger and Tewari. "Sequence Length Independent Norm-Based Generalization Bounds for Transformers." Artificial Intelligence and Statistics, 2024.](https://mlanthology.org/aistats/2024/trauger2024aistats-sequence/)BibTeX
@inproceedings{trauger2024aistats-sequence,
title = {{Sequence Length Independent Norm-Based Generalization Bounds for Transformers}},
author = {Trauger, Jacob and Tewari, Ambuj},
booktitle = {Artificial Intelligence and Statistics},
year = {2024},
pages = {1405-1413},
volume = {238},
url = {https://mlanthology.org/aistats/2024/trauger2024aistats-sequence/}
}