Rethinking Video ViTs: Sparse Video Tubes for Joint Image and Video Learning

Aj Piergiovanni, Weicheng Kuo, Anelia Angelova

CVPR 2023 pp. 2214-2224

doi:10.1109/CVPR52729.2023.00220 /cvpr/2023/piergiovanni2023cvpr-rethinking/

Abstract

We present a simple approach which can turn a ViT encoder into an efficient video model, which can seamlessly work with both image and video inputs. By sparsely sampling the inputs, the model is able to do training and inference from both inputs. The model is easily scalable and can be adapted to large-scale pre-trained ViTs without requiring full finetuning. The model achieves SOTA results.

PDF CVPR Semantic Scholar

Cite

Text

Piergiovanni et al. "Rethinking Video ViTs: Sparse Video Tubes for Joint Image and Video Learning." Conference on Computer Vision and Pattern Recognition, 2023. doi:10.1109/CVPR52729.2023.00220

Markdown

[Piergiovanni et al. "Rethinking Video ViTs: Sparse Video Tubes for Joint Image and Video Learning." Conference on Computer Vision and Pattern Recognition, 2023.](https://mlanthology.org/cvpr/2023/piergiovanni2023cvpr-rethinking/) doi:10.1109/CVPR52729.2023.00220

BibTeX

@inproceedings{piergiovanni2023cvpr-rethinking,
  title     = {{Rethinking Video ViTs: Sparse Video Tubes for Joint Image and Video Learning}},
  author    = {Piergiovanni, Aj and Kuo, Weicheng and Angelova, Anelia},
  booktitle = {Conference on Computer Vision and Pattern Recognition},
  year      = {2023},
  pages     = {2214-2224},
  doi       = {10.1109/CVPR52729.2023.00220},
  url       = {https://mlanthology.org/cvpr/2023/piergiovanni2023cvpr-rethinking/}
}