CLIP-Sculptor: Zero-Shot Generation of High-Fidelity and Diverse Shapes from Natural Language

Aditya Sanghi, Rao Fu, Vivian Liu, Karl D.D. Willis, Hooman Shayani, Amir H. Khasahmadi, Srinath Sridhar, Daniel Ritchie

CVPR 2023 pp. 18339-18348

doi:10.1109/CVPR52729.2023.01759 /cvpr/2023/sanghi2023cvpr-clipsculptor/

Abstract

Recent works have demonstrated that natural language can be used to generate and edit 3D shapes. However, these methods generate shapes with limited fidelity and diversity. We introduce CLIP-Sculptor, a method to address these constraints by producing high-fidelity and diverse 3D shapes without the need for (text, shape) pairs during training. CLIP-Sculptor achieves this in a multi-resolution approach that first generates in a low-dimensional latent space and then upscales to a higher resolution for improved shape fidelity. For improved shape diversity, we use a discrete latent space which is modeled using a transformer conditioned on CLIP's image-text embedding space. We also present a novel variant of classifier-free guidance, which improves the accuracy-diversity trade-off. Finally, we perform extensive experiments demonstrating that CLIP-Sculptor outperforms state-of-the-art baselines.

PDF CVPR Semantic Scholar

Cite

Text

Sanghi et al. "CLIP-Sculptor: Zero-Shot Generation of High-Fidelity and Diverse Shapes from Natural Language." Conference on Computer Vision and Pattern Recognition, 2023. doi:10.1109/CVPR52729.2023.01759

Markdown

[Sanghi et al. "CLIP-Sculptor: Zero-Shot Generation of High-Fidelity and Diverse Shapes from Natural Language." Conference on Computer Vision and Pattern Recognition, 2023.](https://mlanthology.org/cvpr/2023/sanghi2023cvpr-clipsculptor/) doi:10.1109/CVPR52729.2023.01759

BibTeX

@inproceedings{sanghi2023cvpr-clipsculptor,
  title     = {{CLIP-Sculptor: Zero-Shot Generation of High-Fidelity and Diverse Shapes from Natural Language}},
  author    = {Sanghi, Aditya and Fu, Rao and Liu, Vivian and Willis, Karl D.D. and Shayani, Hooman and Khasahmadi, Amir H. and Sridhar, Srinath and Ritchie, Daniel},
  booktitle = {Conference on Computer Vision and Pattern Recognition},
  year      = {2023},
  pages     = {18339-18348},
  doi       = {10.1109/CVPR52729.2023.01759},
  url       = {https://mlanthology.org/cvpr/2023/sanghi2023cvpr-clipsculptor/}
}