In the Eye of Beholder: Joint Learning of Gaze and Actions in First Person Video
Abstract
We address the task of jointly determining what a person is doing and where they are looking based on the analysis of video captured by a headworn camera. We propose a novel deep model for joint gaze estimation and action recognition in First Person Vision. Our method describes the participant's gaze as a probabilistic variable and models its distribution using stochastic units in a deep network. We sample from these stochastic units to generate an attention map. This attention map guides the aggregation of visual features in action recognition, thereby providing coupling between gaze and action. We evaluate our method on the standard EGTEA dataset and demonstrate performance that exceeds the state-of-the-art by a significant margin of 3.5%.
Cite
Text
Li et al. "In the Eye of Beholder: Joint Learning of Gaze and Actions in First Person Video." Proceedings of the European Conference on Computer Vision (ECCV), 2018. doi:10.1007/978-3-030-01228-1_38Markdown
[Li et al. "In the Eye of Beholder: Joint Learning of Gaze and Actions in First Person Video." Proceedings of the European Conference on Computer Vision (ECCV), 2018.](https://mlanthology.org/eccv/2018/li2018eccv-eye/) doi:10.1007/978-3-030-01228-1_38BibTeX
@inproceedings{li2018eccv-eye,
title = {{In the Eye of Beholder: Joint Learning of Gaze and Actions in First Person Video}},
author = {Li, Yin and Liu, Miao and Rehg, James M.},
booktitle = {Proceedings of the European Conference on Computer Vision (ECCV)},
year = {2018},
doi = {10.1007/978-3-030-01228-1_38},
url = {https://mlanthology.org/eccv/2018/li2018eccv-eye/}
}