DATENeRF: Depth-Aware Text-Based Editing of NeRFs
Abstract
Recent diffusion models have demonstrated impressive capabilities for text-based 2D image editing. Applying similar ideas to edit a NeRF scene [?] remains challenging as editing 2D frames individually does not produce multiview-consistent results. We make the key observation that the geometry of a NeRF scene provides a way to unify these 2D edits. We leverage this geometry in depth-conditioned ControlNet [?] to improve the consistency of individual 2D image edits. Furthermore, we propose an inpainting scheme that uses the NeRF scene depth to propagate 2D edits across images while staying robust to errors and resampling issues. We demonstrate that this leads to more consistent, realistic and detailed editing results compared to previous state-of-the-art text-based NeRF editing methods.
Cite
Text
Martinez et al. "DATENeRF: Depth-Aware Text-Based Editing of NeRFs." Proceedings of the European Conference on Computer Vision (ECCV), 2024. doi:10.1007/978-3-031-73247-8_16Markdown
[Martinez et al. "DATENeRF: Depth-Aware Text-Based Editing of NeRFs." Proceedings of the European Conference on Computer Vision (ECCV), 2024.](https://mlanthology.org/eccv/2024/martinez2024eccv-datenerf/) doi:10.1007/978-3-031-73247-8_16BibTeX
@inproceedings{martinez2024eccv-datenerf,
title = {{DATENeRF: Depth-Aware Text-Based Editing of NeRFs}},
author = {Martinez, Sara Rojas and Philip, Julien and Zhang, Kai and Bi, Sai and Luan, Fujun and Ghanem, Bernard and Sunkavalli, Kalyan},
booktitle = {Proceedings of the European Conference on Computer Vision (ECCV)},
year = {2024},
doi = {10.1007/978-3-031-73247-8_16},
url = {https://mlanthology.org/eccv/2024/martinez2024eccv-datenerf/}
}