SHINOBI: Shape and Illumination Using Neural Object Decomposition via BRDF Optimization In-the-Wild

Andreas Engelhardt, Amit Raj, Mark Boss, Yunzhi Zhang, Abhishek Kar, Yuanzhen Li, Deqing Sun, Ricardo Martin Brualla, Jonathan T. Barron, Hendrik P. A. Lensch, Varun Jampani

CVPR 2024 pp. 19636-19646

doi:10.1109/CVPR52733.2024.01857 /cvpr/2024/engelhardt2024cvpr-shinobi/

Abstract

We present SHINOBI an end-to-end framework for the reconstruction of shape material and illumination from object images captured with varying lighting pose and background. Inverse rendering of an object based on unconstrained image collections is a long-standing challenge in computer vision and graphics and requires a joint optimization over shape radiance and pose. We show that an implicit shape representation based on a multi-resolution hash encoding enables faster and robust shape reconstruction with joint camera alignment optimization that outperforms prior work. Further to enable the editing of illumination and object reflectance (i.e. material) we jointly optimize BRDF and illumination together with the object's shape. Our method is class-agnostic and works on in-the-wild image collections of objects to produce relightable 3D assets for several use cases such as AR/VR movies games etc.

PDF CVPR Semantic Scholar

Cite

Text

Engelhardt et al. "SHINOBI: Shape and Illumination Using Neural Object Decomposition via BRDF Optimization In-the-Wild." Conference on Computer Vision and Pattern Recognition, 2024. doi:10.1109/CVPR52733.2024.01857

Markdown

[Engelhardt et al. "SHINOBI: Shape and Illumination Using Neural Object Decomposition via BRDF Optimization In-the-Wild." Conference on Computer Vision and Pattern Recognition, 2024.](https://mlanthology.org/cvpr/2024/engelhardt2024cvpr-shinobi/) doi:10.1109/CVPR52733.2024.01857

BibTeX

@inproceedings{engelhardt2024cvpr-shinobi,
  title     = {{SHINOBI: Shape and Illumination Using Neural Object Decomposition via BRDF Optimization In-the-Wild}},
  author    = {Engelhardt, Andreas and Raj, Amit and Boss, Mark and Zhang, Yunzhi and Kar, Abhishek and Li, Yuanzhen and Sun, Deqing and Brualla, Ricardo Martin and Barron, Jonathan T. and Lensch, Hendrik P. A. and Jampani, Varun},
  booktitle = {Conference on Computer Vision and Pattern Recognition},
  year      = {2024},
  pages     = {19636-19646},
  doi       = {10.1109/CVPR52733.2024.01857},
  url       = {https://mlanthology.org/cvpr/2024/engelhardt2024cvpr-shinobi/}
}