CoDISP: Exploring Compressed Domain Camera ISP with RGB-Guided Encoder

Abstract

Most mobile device Image Signal Processing (ISP) pipelines operate directly on RAW image data for all processing tasks. However, the rise of super-high-resolution cameras on mobile devices has led to increased memory demands for multi-frame ISP pipelines. In this work, we introduce a novel ISP pipeline that operates on a learned compressed domain, aiming to conserve memory for downstream ISP modules’ inputs. We utilize RGB image compression to define a compressed latent domain, preserving both semantic information and high-frequency details. To facilitate mapping of raw images to the compressed domain, we develop a transfer learning strategy. All downstream processing tasks, including demosaicing, single and multi-frame denoising, and registration, are performed on this compressed latent domain. We demonstrate the effectiveness of our compressed domain ISP pipeline on both public and internal datasets. Remarkably, our pipeline achieves ISP performance similar to non-compression methods while significantly reducing mobile memory requirements.