General In-Hand Object Rotation with Vision and Touch
Abstract
We introduce Rotateit, a system that enables fingertip-based object rotation along multiple axes by leveraging multimodal sensory inputs. Our system is trained in simulation, where it has access to ground-truth object shapes and physical properties. Then we distill it to operate on realistic yet noisy simulated visuotactile and proprioceptive sensory inputs. These multimodal inputs are fused via a visuotactile transformer, enabling online inference of object shapes and physical properties during deployment. We show significant performance improvements over prior methods and highlight the importance of visual and tactile sensing.
Cite
Text
Qi et al. "General In-Hand Object Rotation with Vision and Touch." Conference on Robot Learning, 2023.Markdown
[Qi et al. "General In-Hand Object Rotation with Vision and Touch." Conference on Robot Learning, 2023.](https://mlanthology.org/corl/2023/qi2023corl-general/)BibTeX
@inproceedings{qi2023corl-general,
title = {{General In-Hand Object Rotation with Vision and Touch}},
author = {Qi, Haozhi and Yi, Brent and Suresh, Sudharshan and Lambeta, Mike and Ma, Yi and Calandra, Roberto and Malik, Jitendra},
booktitle = {Conference on Robot Learning},
year = {2023},
pages = {2549-2564},
volume = {229},
url = {https://mlanthology.org/corl/2023/qi2023corl-general/}
}