Inferring Social Relations from Visual Concepts

Abstract

In this paper, we study the problem of social relational inference using visual concepts which serve as indicators of actors' social interactions. While social network analysis from videos has started to gain attention in the recent years, the existing work either uses proximity or co-occurrence statistics, or exploit a holistic model of the scene content where the relations are assumed to stay constant throughout the video. This work permits changing relations and argues that there exists a relationship between the visual concepts and the social relations among actors, which is a fundamentally new concept in computer vision. Specifically, we leverage the existing large-scale concept detectors to generate concept score vectors to represent the video content, and we further map them to grouping cues that are used to detect the social structure. In our framework, a probabilistic graphical model with temporal smoothing provides a means to analyze social relations among actors and detect communities. Experiments on Youtube videos and theatrical movies validate the proposed framework.