More Knowledge, Less Bias: Unbiasing Scene Graph Generation with Explicit Ontological Adjustment

Abstract

Scene graph generation (SGG) models seek to detect relationships between objects in a given image. One challenge in this area is the biased distribution of predicates in the dataset and the semantic space. Recent works incorporating knowledge graphs with scene graphs prove effective in improving recall for the tail predicate classes. Moreover, many recent SGG approaches with promising results explicitly redistribute the predicates in both the training process and in the prediction step. To incorporate external knowledge, we construct a commonsense knowledge graph by integrating ConceptNet and Wikidata. To explicitly unbias SGG with knowledge in the reasoning process, we propose a novel framework, Explicit Ontological Adjustment (EOA), to adjust the graph model predictions with knowledge priors. We use the edge matrix from the commonsense knowledge graph as a module in the graph neural network model to refine the relationship detection process. This module proves effective in alleviating the long-tail distribution of predicates. When combined, we show that these modules achieve state-of-the-art performance on the Visual Genome dataset in most cases. The source code will be made publicly available.