Unsupervised Models of Images by Spikeand-Slab RBMs

Abstract

The spike and slab Restricted Boltzmann Machine (RBM) is defined by having both a real valued ``slab'' variable and a binary ``spike'' variable associated with each unit in the hidden layer. In this paper we generalize and extend the spike and slab RBM to include non-zero means of the conditional distribution over the observed variables conditional on the binary spike variables. We also introduce a term, quadratic in the observed data that we exploit to guarantee the all conditionals associated with the model are well defined -- a guarantee that was absent in the original spike and slab RBM. The inclusion of these generalizations improves the performance of the spike and slab RBM as a feature learner and achieves competitive performance on the CIFAR-10 image classification task. The spike and slab model, when trained in a convolutional configuration, can generate sensible samples that demonstrate that the model has capture the broad statistical structure of natural images.