Instruct-NeRF2NeRF: Editing 3D Scenes with Instructions

Ayaan Haque, Matthew Tancik, Alexei A. Efros, Aleksander Holynski, Angjoo Kanazawa

ICCV 2023 pp. 19740-19750

doi:10.1109/ICCV51070.2023.01808 /iccv/2023/haque2023iccv-instructnerf2nerf/

Abstract

We propose a method for editing NeRF scenes with text-instructions. Given a NeRF of a scene and the collection of images used to reconstruct it, our method uses an image-conditioned diffusion model (InstructPix2Pix) to iteratively edit the input images while optimizing the underlying scene, resulting in an optimized 3D scene that respects the edit instruction. We demonstrate that our proposed method is able to edit large-scale, real-world scenes, and is able to accomplish more realistic, targeted edits than prior work.

PDF ICCV Semantic Scholar

Cite

Text

Haque et al. "Instruct-NeRF2NeRF: Editing 3D Scenes with Instructions." International Conference on Computer Vision, 2023. doi:10.1109/ICCV51070.2023.01808

Markdown

[Haque et al. "Instruct-NeRF2NeRF: Editing 3D Scenes with Instructions." International Conference on Computer Vision, 2023.](https://mlanthology.org/iccv/2023/haque2023iccv-instructnerf2nerf/) doi:10.1109/ICCV51070.2023.01808

BibTeX

@inproceedings{haque2023iccv-instructnerf2nerf,
  title     = {{Instruct-NeRF2NeRF: Editing 3D Scenes with Instructions}},
  author    = {Haque, Ayaan and Tancik, Matthew and Efros, Alexei A. and Holynski, Aleksander and Kanazawa, Angjoo},
  booktitle = {International Conference on Computer Vision},
  year      = {2023},
  pages     = {19740-19750},
  doi       = {10.1109/ICCV51070.2023.01808},
  url       = {https://mlanthology.org/iccv/2023/haque2023iccv-instructnerf2nerf/}
}