TieBot: Learning to Knot a Tie from Visual Demonstration Through a Real-to-Sim-to-Real Approach
Abstract
The tie-knotting task is highly challenging due to the tie’s high deformation and long-horizon manipulation actions. This work presents TieBot, a Real-to-Sim-to-Real learning from visual demonstration system for the robots to learn to knot a tie. We introduce the Hierarchical Feature Matching approach to estimate a sequence of tie’s meshes from the demonstration video. With these estimated meshes used as subgoals, we first learn a teacher policy using privileged information. Then, we learn a student policy with point cloud observation by imitating teacher policy. Lastly, our pipeline applies learned policy to real-world execution. We demonstrate the effectiveness of TieBot in simulation and the real world. In the real-world experiment, a dual-arm robot successfully knots a tie, achieving 50% success rate among 10 trials. Videos can be found on https://tiebots.github.io/.
Cite
Text
Peng et al. "TieBot: Learning to Knot a Tie from Visual Demonstration Through a Real-to-Sim-to-Real Approach." Proceedings of The 8th Conference on Robot Learning, 2024.Markdown
[Peng et al. "TieBot: Learning to Knot a Tie from Visual Demonstration Through a Real-to-Sim-to-Real Approach." Proceedings of The 8th Conference on Robot Learning, 2024.](https://mlanthology.org/corl/2024/peng2024corl-tiebot/)BibTeX
@inproceedings{peng2024corl-tiebot,
title = {{TieBot: Learning to Knot a Tie from Visual Demonstration Through a Real-to-Sim-to-Real Approach}},
author = {Peng, Weikun and Lv, Jun and Zeng, Yuwei and Chen, Haonan and Zhao, Siheng and Sun, Jichen and Lu, Cewu and Shao, Lin},
booktitle = {Proceedings of The 8th Conference on Robot Learning},
year = {2024},
pages = {318-339},
volume = {270},
url = {https://mlanthology.org/corl/2024/peng2024corl-tiebot/}
}