Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation

Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A. Yeh, Greg Shakhnarovich

CVPR 2023 pp. 12619-12629

doi:10.1109/CVPR52729.2023.01214 /cvpr/2023/wang2023cvpr-score/

Abstract

A diffusion model learns to predict a vector field of gradients. We propose to apply chain rule on the learned gradients, and back-propagate the score of a diffusion model through the Jacobian of a differentiable renderer, which we instantiate to be a voxel radiance field. This setup aggregates 2D scores at multiple camera viewpoints into a 3D score, and repurposes a pretrained 2D model for 3D data generation. We identify a technical challenge of distribution mismatch that arises in this application, and propose a novel estimation mechanism to resolve it. We run our algorithm on several off-the-shelf diffusion image generative models, including the recently released Stable Diffusion trained on the large-scale LAION dataset.

PDF CVPR Semantic Scholar

Cite

Text

Wang et al. "Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation." Conference on Computer Vision and Pattern Recognition, 2023. doi:10.1109/CVPR52729.2023.01214

Markdown

[Wang et al. "Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation." Conference on Computer Vision and Pattern Recognition, 2023.](https://mlanthology.org/cvpr/2023/wang2023cvpr-score/) doi:10.1109/CVPR52729.2023.01214

BibTeX

@inproceedings{wang2023cvpr-score,
  title     = {{Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation}},
  author    = {Wang, Haochen and Du, Xiaodan and Li, Jiahao and Yeh, Raymond A. and Shakhnarovich, Greg},
  booktitle = {Conference on Computer Vision and Pattern Recognition},
  year      = {2023},
  pages     = {12619-12629},
  doi       = {10.1109/CVPR52729.2023.01214},
  url       = {https://mlanthology.org/cvpr/2023/wang2023cvpr-score/}
}