ML Anthology

InternVideo2: Scaling Foundation Models for Multimodal Video Understanding

Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Jilan Xu, Zun Wang, Yansong Shi, Tianxiang Jiang, SongZe Li, Hongjie Zhang, Yifei Huang, Yu Qiao, Yali Wang, Limin Wang
ECCV 2024
doi:10.1007/978-3-031-73013-9_23 /eccv/2024/wang2024eccv-internvideo2/

Abstract

We introduce , a new family of video foundation models (ViFM) that achieve the state-of-the-art results in video recognition, video-text tasks, and video-centric dialogue. Our core design is a progressive training approach that unifies the masked video modeling, crossmodal contrastive learning, and next token prediction, scaling up the video encoder size to 6B parameters. At the data level, we prioritize spatiotemporal consistency by semantically segmenting videos and generating video-audio-speech captions. This improves the alignment between video and text. Through extensive experiments, we validate our designs and demonstrate superior performance on over 60 video and audio tasks. Notably, our model outperforms others on various video-related dialogue and long video understanding benchmarks, highlighting its ability to reason and comprehend longer contexts. *Equal contribution. †Corresponding authors.

PDF ECCV Semantic Scholar

Cite

Text

Wang et al. "InternVideo2: Scaling Foundation Models for Multimodal Video Understanding." Proceedings of the European Conference on Computer Vision (ECCV), 2024. doi:10.1007/978-3-031-73013-9_23

Markdown

[Wang et al. "InternVideo2: Scaling Foundation Models for Multimodal Video Understanding." Proceedings of the European Conference on Computer Vision (ECCV), 2024.](https://mlanthology.org/eccv/2024/wang2024eccv-internvideo2/) doi:10.1007/978-3-031-73013-9_23

BibTeX

@inproceedings{wang2024eccv-internvideo2,
  title     = {{InternVideo2: Scaling Foundation Models for Multimodal Video Understanding}},
  author    = {Wang, Yi and Li, Kunchang and Li, Xinhao and Yu, Jiashuo and He, Yinan and Chen, Guo and Pei, Baoqi and Zheng, Rongkun and Xu, Jilan and Wang, Zun and Shi, Yansong and Jiang, Tianxiang and Li, SongZe and Zhang, Hongjie and Huang, Yifei and Qiao, Yu and Wang, Yali and Wang, Limin},
  booktitle = {Proceedings of the European Conference on Computer Vision (ECCV)},
  year      = {2024},
  doi       = {10.1007/978-3-031-73013-9_23},
  url       = {https://mlanthology.org/eccv/2024/wang2024eccv-internvideo2/}
}