Cloth-Splatting: 3D Cloth State Estimation from RGB Supervision
Abstract
We introduce Cloth-Splatting, a method for estimating 3D states of cloth from RGB images through a prediction-update framework. Cloth-Splatting leverages an action-conditioned dynamics model for predicting future states and uses 3D Gaussian Splatting to update the predicted states. Our key insight is that coupling a 3D mesh-based representation with Gaussian Splatting allows us to define a differentiable map between the cloth’s state space and the image space. This enables the use of gradient-based optimization techniques to refine inaccurate state estimates using only RGB supervision. Our experiments demonstrate that Cloth-Splatting not only improves state estimation accuracy over current baselines but also reduces convergence time by $\sim 85$ %.
Cite
Text
Longhini et al. "Cloth-Splatting: 3D Cloth State Estimation from RGB Supervision." Proceedings of The 8th Conference on Robot Learning, 2024.Markdown
[Longhini et al. "Cloth-Splatting: 3D Cloth State Estimation from RGB Supervision." Proceedings of The 8th Conference on Robot Learning, 2024.](https://mlanthology.org/corl/2024/longhini2024corl-clothsplatting/)BibTeX
@inproceedings{longhini2024corl-clothsplatting,
title = {{Cloth-Splatting: 3D Cloth State Estimation from RGB Supervision}},
author = {Longhini, Alberta and Büsching, Marcel and Duisterhof, Bardienus Pieter and Lundell, Jens and Ichnowski, Jeffrey and Björkman, Mårten and Kragic, Danica},
booktitle = {Proceedings of The 8th Conference on Robot Learning},
year = {2024},
pages = {2845-2865},
volume = {270},
url = {https://mlanthology.org/corl/2024/longhini2024corl-clothsplatting/}
}