Open-Vocabulary 3D Semantic Segmentation with Text-to-Image Diffusion Models
Abstract
In this paper, we investigate the use of diffusion models which are pre-trained on large-scale image-caption pairs for open-vocabulary 3D semantic understanding. We propose a novel method, namely Diff2Scene, which leverages frozen representations from text-image generative models, along with salient-aware and geometric-aware masks, for open-vocabulary 3D semantic segmentation and visual grounding tasks. Diff2Scene gets rid of any labeled 3D data and effectively identifies objects, appearances, materials, locations and their compositions in 3D scenes. We show that it outperforms competitive baselines and achieves significant improvements over state-of-the-art methods. In particular, Diff2Scene improves the state-of-the-art method on ScanNet200 by 12%.
Cite
Text
Zhu et al. "Open-Vocabulary 3D Semantic Segmentation with Text-to-Image Diffusion Models." Proceedings of the European Conference on Computer Vision (ECCV), 2024. doi:10.1007/978-3-031-73397-0_21Markdown
[Zhu et al. "Open-Vocabulary 3D Semantic Segmentation with Text-to-Image Diffusion Models." Proceedings of the European Conference on Computer Vision (ECCV), 2024.](https://mlanthology.org/eccv/2024/zhu2024eccv-openvocabulary/) doi:10.1007/978-3-031-73397-0_21BibTeX
@inproceedings{zhu2024eccv-openvocabulary,
title = {{Open-Vocabulary 3D Semantic Segmentation with Text-to-Image Diffusion Models}},
author = {Zhu, Xiaoyu and Zhou, Hao and Xing, Pengfei and Zhao, Long and Xu, Hao and Liang, Junwei and Hauptmann, Alexander G. and Liu, Ting and Gallagher, Andrew},
booktitle = {Proceedings of the European Conference on Computer Vision (ECCV)},
year = {2024},
doi = {10.1007/978-3-031-73397-0_21},
url = {https://mlanthology.org/eccv/2024/zhu2024eccv-openvocabulary/}
}