Codec Avatar Studio: Paired Human Captures for Complete, Driveable, and Generalizable Avatars
Abstract
To build photorealistic avatars that users can embody, human modelling must be complete (cover the full body), driveable (able to reproduce the current motion and appearance from the user), and generalizable (i.e., easily adaptable to novel identities).Towards these goals, paired captures, that is, captures of the same subject obtained from systems of diverse quality and availability, are crucial.However, paired captures are rarely available to researchers outside of dedicated industrial labs: Codec Avatar Studio is our proposal to close this gap.Towards generalization and driveability, we introduce a dataset of 256 subjects captured in two modalities: high resolution multi-view scans of their heads, and video from the internal cameras of a headset.Towards completeness, we introduce a dataset of 4 subjects captured in eight modalities: high quality relightable multi-view captures of heads and hands, full body multi-view captures with minimal and regular clothes, and corresponding head, hands and body phone captures.Together with our data, we also provide code and pre-trained models for different state-of-the-art human generation models.Our datasets and code are available at https://github.com/facebookresearch/ava-256 and https://github.com/facebookresearch/goliath.
Cite
Text
Martinez et al. "Codec Avatar Studio: Paired Human Captures for Complete, Driveable, and Generalizable Avatars." Neural Information Processing Systems, 2024. doi:10.52202/079017-2640Markdown
[Martinez et al. "Codec Avatar Studio: Paired Human Captures for Complete, Driveable, and Generalizable Avatars." Neural Information Processing Systems, 2024.](https://mlanthology.org/neurips/2024/martinez2024neurips-codec/) doi:10.52202/079017-2640BibTeX
@inproceedings{martinez2024neurips-codec,
title = {{Codec Avatar Studio: Paired Human Captures for Complete, Driveable, and Generalizable Avatars}},
author = {Martinez, Julieta and Kim, Emily and Romero, Javier and Bagautdinov, Timur and Saito, Shunsuke and Yu, Shoou-I and Anderson, Stuart and Zollhöfer, Michael and Wang, Te-Li and Bai, Shaojie and Li, Chenghui and Wei, Shih-En and Joshi, Rohan and Borsos, Wyatt and Simon, Tomas and Saragih, Jason and Theodosis, Paul and Greene, Alexander and Josyula, Anjani and Maeta, Silvio Mano and Jewett, Andrew I. and Venshtain, Simon and Heilman, Christopher and Chen, Yueh-Tung and Fu, Sidi and Elshaer, Mohamed Ezzeldin A. and Du, Tingfang and Wu, Longhua and Chen, Shen-Chi and Kang, Kai and Wu, Michael and Emad, Youssef and Longay, Steven and Brewer, Ashley and Shah, Hitesh and Booth, James and Koska, Taylor and Haidle, Kayla and Andromalos, Matt and Hsu, Joanna and Dauer, Thomas and Selednik, Peter and Godisart, Tim and Ardisson, Scott and Cipperly, Matthew and Humberston, Ben and Farr, Lon and Hansen, Bob and Guo, Peihong and Braun, Dave and Krenn, Steven and Wen, He and Evans, Lucas and Fadeeva, Natalia and Stewart, Matthew and Schwartz, Gabriel and Gupta, Divam and Moon, Gyeongsik and Guo, Kaiwen and Dong, Yuan and Xu, Yichen and Shiratori, Takaaki and Prada, Fabian and Pires, Bernardo R. and Peng, Bo and Buffalini, Julia and Trimble, Autumn and McPhail, Kevyn and Schoeller, Melissa and Sheikh, Yaser},
booktitle = {Neural Information Processing Systems},
year = {2024},
doi = {10.52202/079017-2640},
url = {https://mlanthology.org/neurips/2024/martinez2024neurips-codec/}
}