NeRFDeformer: NeRF Transformation from a Single View via 3D Scene Flows
Abstract
We present a method for automatically modifying a NeRF representation based on a single observation of a non-rigid transformed version of the original scene. Our method defines the transformation as a 3D flowspecifically as a weighted linear blending of rigid transformations of 3D anchor points that are defined on the surface of the scene. In order to identify anchor points we introduce a novel correspondence algorithm that first matches RGB-based pairs then leverages multi-view information and 3D reprojection to robustly filter false positives in two steps. We also introduce a new dataset for exploring the problem of modifying a NeRF scene through a single observation. Our dataset contains 113 scenes leveraging 47 3D assets.We show that our proposed method outperforms NeRF editing methods as well as diffusion-based methods and we also explore different methods for filtering correspondences.
Cite
Text
Tang et al. "NeRFDeformer: NeRF Transformation from a Single View via 3D Scene Flows." Conference on Computer Vision and Pattern Recognition, 2024. doi:10.1109/CVPR52733.2024.00980Markdown
[Tang et al. "NeRFDeformer: NeRF Transformation from a Single View via 3D Scene Flows." Conference on Computer Vision and Pattern Recognition, 2024.](https://mlanthology.org/cvpr/2024/tang2024cvpr-nerfdeformer/) doi:10.1109/CVPR52733.2024.00980BibTeX
@inproceedings{tang2024cvpr-nerfdeformer,
title = {{NeRFDeformer: NeRF Transformation from a Single View via 3D Scene Flows}},
author = {Tang, Zhenggang and Ren, Zhongzheng and Zhao, Xiaoming and Wen, Bowen and Tremblay, Jonathan and Birchfield, Stan and Schwing, Alexander},
booktitle = {Conference on Computer Vision and Pattern Recognition},
year = {2024},
pages = {10293-10303},
doi = {10.1109/CVPR52733.2024.00980},
url = {https://mlanthology.org/cvpr/2024/tang2024cvpr-nerfdeformer/}
}