Conditional Hallucinations for Image Compression

Abstract

In lossy image compression, models face the challenge of either hallucinating details or generating out-of-distribution samples due to the information bottleneck. This implies that at times, introducing hallucinations is necessary to generate in-distribution samples. The optimal level of hallucination varies depending on image content, as humans are sensitive to small changes that alter the semantic meaning. We propose a novel compression method that dynamically balances the degree of hallucination based on content. We collect data and train a model to predict user preferences on hallucinations. By using this prediction to adjust the perceptual weight in the reconstruction loss, we develop a \textbf{Con}ditionally \textbf{Ha}llucinating compression model (\textbf{ConHa}) that outperforms state-of-the-art image compression methods. Code and images are available at \href{https://polybox.ethz.ch/index.php/s/owS1k5JYs4KD4TA}https://polybox.ethz.ch/index.php/s/owS1k5JYs4KD4TA.