AssistQ: Affordance-Centric Question-Driven Task Completion for Egocentric Assistant
Abstract
A long-standing goal of intelligent assistants such as AR glasses/robots has been to assist users in affordance-centric real-world scenarios, such as ""how can I run the microwave for 1 minute?”. However, there is still no clear task definition and suitable benchmarks. In this paper, we define a new task called Affordance-centric Question-driven Task Completion, where the AI assistant should learn from instructional videos to provide step-by-step help in the user’s view. To support the task, we constructed AssistQ, a new dataset comprising 531 question-answer samples from 100 newly filmed instructional videos. We also developed a novel Question-to-Actions (Q2A) model to address the AQTC task and validate it on the AssistQ dataset. The results show that our model significantly outperforms several VQA-related baselines while still having large room for improvement. We expect our task and dataset to advance Egocentric AI Assistant’s development. Our project page is available at: https://showlab.github.io/assistq/.
Cite
Text
Wong et al. "AssistQ: Affordance-Centric Question-Driven Task Completion for Egocentric Assistant." Proceedings of the European Conference on Computer Vision (ECCV), 2022. doi:10.1007/978-3-031-20059-5_28Markdown
[Wong et al. "AssistQ: Affordance-Centric Question-Driven Task Completion for Egocentric Assistant." Proceedings of the European Conference on Computer Vision (ECCV), 2022.](https://mlanthology.org/eccv/2022/wong2022eccv-assistq/) doi:10.1007/978-3-031-20059-5_28BibTeX
@inproceedings{wong2022eccv-assistq,
title = {{AssistQ: Affordance-Centric Question-Driven Task Completion for Egocentric Assistant}},
author = {Wong, Benita and Chen, Joya and Wu, You and Lei, Stan Weixian and Mao, Dongxing and Gao, Difei and Shou, Mike Zheng},
booktitle = {Proceedings of the European Conference on Computer Vision (ECCV)},
year = {2022},
doi = {10.1007/978-3-031-20059-5_28},
url = {https://mlanthology.org/eccv/2022/wong2022eccv-assistq/}
}