Low Bandwidth Video-Chat Compression Using Deep Generative Models

Maxime Oquab, Pierre Stock, Daniel Haziza, Tao Xu, Peizhao Zhang, Onur Celebi, Yana Hasson, Patrick Labatut, Bobo Bose-Kolanu, Thibault Peyronel, Camille Couprie

CVPRW 2021 pp. 2388-2397

doi:10.1109/CVPRW53098.2021.00271 /cvprw/2021/oquab2021cvprw-low/

Abstract

To unlock video chat for hundreds of millions of people hindered by poor connectivity or unaffordable data costs, we propose to authentically reconstruct faces on the receiver’s device using facial landmarks extracted at the sender’s side and transmitted over the network. In this context, we discuss and evaluate the benefits and disadvantages of several deep adversarial approaches. In particular, we explore quality and bandwidth trade-offs for approaches based on static landmarks, dynamic landmarks or segmentation maps. We design a mobile-compatible architecture based on the first order animation model of Siarohin et al. In addition, we leverage SPADE blocks to refine results in important areas such as the eyes and lips. We compress the networks down to about 3 MB, allowing models to run in real time on iPhone 8 (CPU). This approach enables video calling at a few kbits per second, an order of magnitude lower than currently available alternatives.

PDF CVPRW Semantic Scholar

Cite

Text

Oquab et al. "Low Bandwidth Video-Chat Compression Using Deep Generative Models." IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2021. doi:10.1109/CVPRW53098.2021.00271

Markdown

[Oquab et al. "Low Bandwidth Video-Chat Compression Using Deep Generative Models." IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2021.](https://mlanthology.org/cvprw/2021/oquab2021cvprw-low/) doi:10.1109/CVPRW53098.2021.00271

BibTeX

@inproceedings{oquab2021cvprw-low,
  title     = {{Low Bandwidth Video-Chat Compression Using Deep Generative Models}},
  author    = {Oquab, Maxime and Stock, Pierre and Haziza, Daniel and Xu, Tao and Zhang, Peizhao and Celebi, Onur and Hasson, Yana and Labatut, Patrick and Bose-Kolanu, Bobo and Peyronel, Thibault and Couprie, Camille},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops},
  year      = {2021},
  pages     = {2388-2397},
  doi       = {10.1109/CVPRW53098.2021.00271},
  url       = {https://mlanthology.org/cvprw/2021/oquab2021cvprw-low/}
}