Hiera: A Hierarchical Vision Transformer Without the Bells-and-Whistles
Abstract
Modern hierarchical vision transformers have added several vision-specific components in the pursuit of supervised classification performance. While these components lead to effective accuracies and attractive FLOP counts, the added complexity actually makes these transformers slower than their vanilla ViT counterparts. In this paper, we argue that this additional bulk is unnecessary. By pretraining with a strong visual pretext task (MAE), we can strip out all the bells-and-whistles from a state-of-the-art multi-stage vision transformer without losing accuracy. In the process, we create Hiera, an extremely simple hierarchical vision transformer that is more accurate than previous models while being significantly faster both at inference and during training. We evaluate Hiera on a variety of tasks for image and video recognition. Our code and models are available at https://github.com/facebookresearch/hiera.
Cite
Text
Ryali et al. "Hiera: A Hierarchical Vision Transformer Without the Bells-and-Whistles." International Conference on Machine Learning, 2023.Markdown
[Ryali et al. "Hiera: A Hierarchical Vision Transformer Without the Bells-and-Whistles." International Conference on Machine Learning, 2023.](https://mlanthology.org/icml/2023/ryali2023icml-hiera/)BibTeX
@inproceedings{ryali2023icml-hiera,
title = {{Hiera: A Hierarchical Vision Transformer Without the Bells-and-Whistles}},
author = {Ryali, Chaitanya and Hu, Yuan-Ting and Bolya, Daniel and Wei, Chen and Fan, Haoqi and Huang, Po-Yao and Aggarwal, Vaibhav and Chowdhury, Arkabandhu and Poursaeed, Omid and Hoffman, Judy and Malik, Jitendra and Li, Yanghao and Feichtenhofer, Christoph},
booktitle = {International Conference on Machine Learning},
year = {2023},
pages = {29441-29454},
volume = {202},
url = {https://mlanthology.org/icml/2023/ryali2023icml-hiera/}
}