ProTransformer: Robustify Transformers via Plug-and-Play Paradigm
Abstract
Transformer-based architectures have dominated many machine learning areas in recent years. In this paper, we propose a simple yet highly effective robust attention mechanism to robustify any transformer-based architectures. Our algorithm can be implemented with only 4 lines of code and be plugged into any given transformer as a plug-and-play layer to enhance its robustness without additional training or fine-tuning. Comprehensive experiments and ablation studies show that the proposed ProTransformer significantly improves the robustness across various prediction tasks, attack mechanisms, backbone architectures, and data domains.
Cite
Text
Hou et al. "ProTransformer: Robustify Transformers via Plug-and-Play Paradigm." ICLR 2024 Workshops: R2-FM, 2024.Markdown
[Hou et al. "ProTransformer: Robustify Transformers via Plug-and-Play Paradigm." ICLR 2024 Workshops: R2-FM, 2024.](https://mlanthology.org/iclrw/2024/hou2024iclrw-protransformer/)BibTeX
@inproceedings{hou2024iclrw-protransformer,
title = {{ProTransformer: Robustify Transformers via Plug-and-Play Paradigm}},
author = {Hou, Zhichao and Gao, Weizhi and Shen, Yuchen and Liu, Xiaorui},
booktitle = {ICLR 2024 Workshops: R2-FM},
year = {2024},
url = {https://mlanthology.org/iclrw/2024/hou2024iclrw-protransformer/}
}