End-to-End Learning of Visual Representations from Uncurated Instructional Videos
Abstract
Annotating videos is cumbersome, expensive and not scalable. Yet, many strong video models still rely on manually annotated data. With the recent introduction of the HowTo100M dataset, narrated videos now offer the possibility of learning video representations without manual supervision. In this work we propose a new learning approach, MIL-NCE, capable of addressing mis- alignments inherent in narrated videos. With this approach we are able to learn strong video representations from scratch, without the need for any manual annotation. We evaluate our representations on a wide range of four downstream tasks over eight datasets: action recognition (HMDB-51, UCF-101, Kinetics-700), text-to- video retrieval (YouCook2, MSR-VTT), action localization (YouTube-8M Segments, CrossTask) and action segmentation (COIN). Our method outperforms all published self-supervised approaches for these tasks as well as several fully supervised baselines.
Cite
Text
Miech et al. "End-to-End Learning of Visual Representations from Uncurated Instructional Videos." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020. doi:10.1109/CVPR42600.2020.00990Markdown
[Miech et al. "End-to-End Learning of Visual Representations from Uncurated Instructional Videos." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.](https://mlanthology.org/cvpr/2020/miech2020cvpr-endtoend/) doi:10.1109/CVPR42600.2020.00990BibTeX
@inproceedings{miech2020cvpr-endtoend,
title = {{End-to-End Learning of Visual Representations from Uncurated Instructional Videos}},
author = {Miech, Antoine and Alayrac, Jean-Baptiste and Smaira, Lucas and Laptev, Ivan and Sivic, Josef and Zisserman, Andrew},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
year = {2020},
doi = {10.1109/CVPR42600.2020.00990},
url = {https://mlanthology.org/cvpr/2020/miech2020cvpr-endtoend/}
}