Towards Regression-Free Neural Networks for Diverse Compute Platforms

Abstract

With the shift towards on-device deep learning, ensuring a consistent behavior of an AI service across diverse compute platforms becomes tremendously important. Our work tackles the emergent problem of reducing predictive in-consistencies arising as negative flips: test samples that are correctly predicted by a less accurate on-device model, but incorrectly by a more accurate on-cloud one. We introduce REGression constrained Neural Architecture Search (REG-NAS) to design a family of highly accurate models that engender fewer negative flips. REG-NAS consists of two components: (1) A novel architecture constraint that enables a larger on-cloud model to contain all the weights of the smaller on-device one thus maximizing weight sharing. This idea stems from our observation that larger weight sharing among networks leads to similar sample-wise predictions and results in fewer negative flips; (2) A novel search reward that incorporates both Top-1 accuracy and negative flips in the architecture optimization metric. We demonstrate that REG-NAS can successfully find architecture with few negative flips, in three popular architecture search spaces. Compared to the existing state-of-the-art approach [29], REG-NAS leads to 33-48% relative reduction of negative flips.