Breaking the Likelihood--Quality Trade-Off in Diffusion Models by Merging Pretrained Experts

Abstract

Diffusion models have recently emerged as powerful generative models capable of producing highly realistic images. Despite their success, a persistent challenge remains: models that generate high-quality samples often assign poor likelihoods to data, and vice versa. This trade-off arises because perceptual quality depends more on modeling high-noise regions, while likelihood is dominated by sensitivity to low-level image statistics. In this work, we propose a simple yet effective method to overcome this trade-off by merging two pretrained diffusion experts, one focused on perceptual quality and the other on likelihood, within a Mixture-of-Experts framework. Our approach applies the image-quality expert during high noise levels and uses the likelihood expert in low noise levels. Empirically, our merged model consistently improves over both experts: on CIFAR-10, it achieves better likelihood and sample quality than either baseline. On ImageNet32, it matches the likelihood of the likelihood expert while surpassing the image-quality expert in FID, effectively breaking the likelihood–quality trade-off in diffusion models.