Statistically-Motivated Second-Order Pooling

Abstract

However, the nature of such operations is usually computationally expensive, and resulting vector representation orders of magnitude larger than first-order baselines. Here, by contrast, we introduce a statistically-motivated framework that projects the second-order descriptor into a compact vector while improving the representational power. To this end, we design a parametric vectorization layer, which maps a covariance matrix, known to follow a Wishart distribution, to a vector whose elements can be shown to follow a Chi-square distribution. We then propose to make use of a square-root normalization, which makes the distribution of the resulting representation converge to a Gaussian, with which most classifiers of recent first-order networks complying. As evidenced by our experiments, this lets us outperform the state-of-the-art first-order and second-order models on several benchmark recognition datasets.