The Trichromatic Strong Lottery Ticket Hypothesis: Neural Compression with Three Primary Supermasks

Abstract

The Strong Lottery Ticket Hypothesis (SLTH) demonstrated that a high-performing model can be obtained just by pruning a randomly initialized dense neural network by optimizing a pruning mask, known as a supermask. Supermask accuracy has recently been enhanced by incorporating sign flipping or weight scaling. Furthermore, it has been demonstrated that supermask training can be extended to sparse random networks. This work proposes the Trichromatic Strong Lottery Hypothesis (T-SLTH), a generalization of the SLTH that (1) connects supermasks to quantization-aware training, (2) consolidates all existing supermasks into a single design framework based on three additive primary supermasks, and (3) contains novel supermask types that support arbitrary connectivity. In addition to sparsity and quantization, the partial randomness of supermask-based models provides specialized digital hardware accelerators with a unique opportunity for neural compression. The models offered by the T-SLTH set the SoTA for supermask-based models in accuracy-size tradeoff: a ResNet-$50$ scoring $78.43$% on CIFAR-$100$ can be compressed $38\times$ to $2.51$ MB, or even $144\times$ down to $0.66$ MB while retaining $74.52$% accuracy, and $25\times$ to $4.1$ MB while scoring $75.28$% on ImageNet.