ML Anthology

MOSAIC: Modular Foundation Models for Assistive and Interactive Cooking

Huaxiaoyue Wang, Kushal Kedia, Juntao Ren, Rahma Abdullah, Atiksh Bhardwaj, Angela Chao, Kelly Y Chen, Nathaniel Chin, Prithwish Dan, Xinyi Fan, Gonzalo Gonzalez-Pumariega, Aditya Kompella, Maximus Adrian Pace, Yash Sharma, Xiangwan Sun, Neha Sunkara, Sanjiban Choudhury
CoRL 2024 pp. 2220-2294
/corl/2024/wang2024corl-mosaic/

Abstract

We present MOSAIC, a modular architecture for coordinating multiple robots to (a) interact with users using natural language and (b) manipulate an open vocabulary of everyday objects. At several levels, MOSAIC employs modularity: it leverages multiple large-scale pre-trained models for high-level tasks like language and image recognition, while using streamlined modules designed for low-level task-specific control. This decomposition allows us to reap the complementary benefits of foundation models and precise, more specialized models, enabling our system to scale to complex tasks that involve coordinating multiple robots and humans. First, we unit-test individual modules with 180 episodes of visuomotor picking, 60 episodes of human motion forecasting, and 46 online user evaluations of the task planner. We then extensively evaluate MOSAIC with 60 end-to-end trials. We discuss crucial design decisions, limitations of the current system, and open challenges in this domain

PDF CoRL OpenReview Semantic Scholar

Cite

Text

Wang et al. "MOSAIC: Modular Foundation Models for Assistive and Interactive Cooking." Proceedings of The 8th Conference on Robot Learning, 2024.

Markdown

[Wang et al. "MOSAIC: Modular Foundation Models for Assistive and Interactive Cooking." Proceedings of The 8th Conference on Robot Learning, 2024.](https://mlanthology.org/corl/2024/wang2024corl-mosaic/)

BibTeX

@inproceedings{wang2024corl-mosaic,
  title     = {{MOSAIC: Modular Foundation Models for Assistive and Interactive Cooking}},
  author    = {Wang, Huaxiaoyue and Kedia, Kushal and Ren, Juntao and Abdullah, Rahma and Bhardwaj, Atiksh and Chao, Angela and Chen, Kelly Y and Chin, Nathaniel and Dan, Prithwish and Fan, Xinyi and Gonzalez-Pumariega, Gonzalo and Kompella, Aditya and Pace, Maximus Adrian and Sharma, Yash and Sun, Xiangwan and Sunkara, Neha and Choudhury, Sanjiban},
  booktitle = {Proceedings of The 8th Conference on Robot Learning},
  year      = {2024},
  pages     = {2220-2294},
  volume    = {270},
  url       = {https://mlanthology.org/corl/2024/wang2024corl-mosaic/}
}