Recognizing Unseen States of Unknown Objects by Leveraging Knowledge Graphs

Abstract

We investigate the problem of Object State Classification (OSC) in the context of zero-shot learning. Specifically we propose the first method for Zero-shot Object-agnostic State Classification (OaSC) that given an image infers the state of a single object without relying on the knowledge or the estimation of the object class. In that direction we capitalize on Knowledge Graphs (KGs) for structuring and organizing external knowledge which in combination with visual information enable effective inference of the states of objects that have not been encountered in the training set. Having this unique property a significant strength of our method is that it can handle an Open Set of object classes. We investigate the performance of OaSC in various datasets and settings against several hypotheses and in comparison with state-of-the-art approaches for object attribute classification. OaSC outperforms these methods significantly across all benchmarks.