ProGBA: Prompt Guided Bayesian Augmentation for Zero-Shot Domain Adaptation

Abstract

Domain adaptation is a well-established field within computer vision. Due to the common scenario of inaccessible target domain data, zero-shot domain adaptation increasingly gets more attention. Existing methods, which primarily focus on optimizing an Empirical Risk Minimization objective, tend to rely on training with discrete augmentations based on limited prompts. This strategy struggles to fully capture the complexity of the target domain, consequently diminishing the transferred model’s effectiveness. In this paper, we introduce ProGBA, a novel framework that adopts a Bayesian perspective to regard the learning process in zero-shot domain adaptation as a variational inference problem. This approach aims to comprehend the distribution of domain-adaptive augmentations. Leveraging Bayesian methods’ regularization capabilities, ProGBA refines the domain adaptation representation space, which helps to mitigate the overfitting risks. Specifically, ProGBA adeptly introduces the uncertainties associated with domain shifts through probabilistic modeling of residuals between the source and target domains, which reduces the model’s reliance on a specific set of weights, thereby enhancing performance in the target domain. Furthermore, we adopt a pre-trained visual-language model alongside a novel text-based loss function to more accurately align the learned distribution with the actual residual distribution between the target and source domains. The comprehensive validation showcases ProGBA’s potential to set a new benchmark in zero-shot domain adaptation, demonstrating ProGBA’s efficacy in adapting to the target domain. Moreover, extensive experiments on cross-domain semantic segmentation also underscore our method’s generalizability.