Gaze Embeddings for Zero-Shot Image Classification
Abstract
Zero-shot image classification using auxiliary information, such as attributes describing discriminative object properties, requires time-consuming annotation by domain experts. We instead propose a method that relies on human gaze as auxiliary information, exploiting that even non-expert users have a natural ability to judge class membership. We present a data collection paradigm that involves a discrimination task to increase the information content obtained from gaze data. Our method extracts discriminative descriptors from the data and learns a compatibility function between image and gaze using three novel gaze embeddings: Gaze Histograms (GH), Gaze Features with Grid (GFG) and Gaze Features with Sequence (GFS). We introduce two new gaze-annotated datasets for fine-grained image classification and show that human gaze data is indeed class discriminative, provides a competitive alternative to expert-annotated attributes, and outperforms other baselines for zero-shot image classification.
Cite
Text
Karessli et al. "Gaze Embeddings for Zero-Shot Image Classification." Conference on Computer Vision and Pattern Recognition, 2017. doi:10.1109/CVPR.2017.679Markdown
[Karessli et al. "Gaze Embeddings for Zero-Shot Image Classification." Conference on Computer Vision and Pattern Recognition, 2017.](https://mlanthology.org/cvpr/2017/karessli2017cvpr-gaze/) doi:10.1109/CVPR.2017.679BibTeX
@inproceedings{karessli2017cvpr-gaze,
title = {{Gaze Embeddings for Zero-Shot Image Classification}},
author = {Karessli, Nour and Akata, Zeynep and Schiele, Bernt and Bulling, Andreas},
booktitle = {Conference on Computer Vision and Pattern Recognition},
year = {2017},
doi = {10.1109/CVPR.2017.679},
url = {https://mlanthology.org/cvpr/2017/karessli2017cvpr-gaze/}
}