Dense Hopfield Networks with Hierarchical Memories

Abstract

We consider a 3-level hierarchical generative model for memories which are sampled and stored in a dense Hopfield network with polynomial activation. We analytically derive conditions for each level of this hierarchy to be locally stable -- that is they are local energy maxima. We find that it takes only a polynomial amount of information to generalize beyond particular memories and even particular groups in the hierarchy. Our theory predicts the qualitative features a phase diagram in the number of memories, sharpness of the activation function (polynomial degree) for data from Fashion-MNIST.