Painting 3D Nature in 2D: View Synthesis of Natural Scenes from a Single Semantic Mask
Abstract
We introduce a novel approach that takes a single semantic mask as input to synthesize multi-view consistent color images of natural scenes, trained with a collection of single images from the Internet. Prior works on 3D-aware image synthesis either require multi-view supervision or learning category-level prior for specific classes of objects, which are inapplicable to natural scenes. Our key idea to solve this challenge is to use a semantic field as the intermediate representation, which is easier to reconstruct from an input semantic mask and then translated to a radiance field with the assistance of off-the-shelf semantic image synthesis models. Experiments show that our method outperforms baseline methods and produces photorealistic and multi-view consistent videos of a variety of natural scenes. The project website is https://zju3dv.github.io/paintingnature/.
Cite
Text
Zhang et al. "Painting 3D Nature in 2D: View Synthesis of Natural Scenes from a Single Semantic Mask." Conference on Computer Vision and Pattern Recognition, 2023. doi:10.1109/CVPR52729.2023.00823Markdown
[Zhang et al. "Painting 3D Nature in 2D: View Synthesis of Natural Scenes from a Single Semantic Mask." Conference on Computer Vision and Pattern Recognition, 2023.](https://mlanthology.org/cvpr/2023/zhang2023cvpr-painting/) doi:10.1109/CVPR52729.2023.00823BibTeX
@inproceedings{zhang2023cvpr-painting,
title = {{Painting 3D Nature in 2D: View Synthesis of Natural Scenes from a Single Semantic Mask}},
author = {Zhang, Shangzhan and Peng, Sida and Chen, Tianrun and Mou, Linzhan and Lin, Haotong and Yu, Kaicheng and Liao, Yiyi and Zhou, Xiaowei},
booktitle = {Conference on Computer Vision and Pattern Recognition},
year = {2023},
pages = {8518-8528},
doi = {10.1109/CVPR52729.2023.00823},
url = {https://mlanthology.org/cvpr/2023/zhang2023cvpr-painting/}
}