Physically Plausible Spectral Reconstruction from RGB Images

Abstract

Recently Convolutional Neural Networks (CNN) have been used to reconstruct hyperspectral information from RGB images, and this spectral reconstruction problem (SR) can often be solved with good (low) error. However, little attention has been paid on whether these models’ behavior can adhere to physics. We show that the leading CNN method introduces unexpected ‘colorimetric errors’, which means the recovered spectra do not reproduce ground-truth RGBs, and sometimes this discrepancy can be large. The problem is further compounded by exposure change. Indeed, most CNN models over-fit to fixed exposure and we demonstrate that this can result in poor performance when exposure varies.In this paper we show how CNN learning can be extended so that the physical plausibility of SR is enforced. Remarkably, our physically plausible CNN solutions advance both spectral and colorimetric performance of the original network, while the application of data augmentation trades off the network performance for model stability against varying exposure.