Self-Supervised Learning of Audio-Visual Objects from Video
Abstract
Our objective is to transform a video into a set of discrete audio-visual objects using self-supervised learning. To this end we introduce a model that uses attention to localize and group sound sources, and optical flow to aggregate information over time. We demonstrate the effectiveness of the audio-visual object embeddings that our model learns by using them for four downstream speech-oriented tasks: (a) multi-speaker sound source separation, (b) localizing and tracking speakers, (c) correcting misaligned audio-visual data, and (d) active speaker detection. Using our representation, these tasks can be solved entirely by training on unlabeled video, without the aid of object detectors. We also demonstrate the generality of our method by applying it to non-human speakers, including cartoons and puppets. Our model significantly outperforms other self-supervised approaches, and obtains performance competitive with methods that use supervised face detection.
Cite
Text
Afouras et al. "Self-Supervised Learning of Audio-Visual Objects from Video." Proceedings of the European Conference on Computer Vision (ECCV), 2020. doi:10.1007/978-3-030-58523-5_13Markdown
[Afouras et al. "Self-Supervised Learning of Audio-Visual Objects from Video." Proceedings of the European Conference on Computer Vision (ECCV), 2020.](https://mlanthology.org/eccv/2020/afouras2020eccv-selfsupervised/) doi:10.1007/978-3-030-58523-5_13BibTeX
@inproceedings{afouras2020eccv-selfsupervised,
title = {{Self-Supervised Learning of Audio-Visual Objects from Video}},
author = {Afouras, Triantafyllos and Owens, Andrew and Chung, Joon Son and Zisserman, Andrew},
booktitle = {Proceedings of the European Conference on Computer Vision (ECCV)},
year = {2020},
doi = {10.1007/978-3-030-58523-5_13},
url = {https://mlanthology.org/eccv/2020/afouras2020eccv-selfsupervised/}
}