Bayesian Low-Rank Learning (Bella): A Practical Approach to Bayesian Neural Networks

Abstract

Computational complexity of Bayesian learning is impeding its adoption in practical, large-scale tasks, despite demonstrations of significant merits such as improved robustness and resilience to unseen or out-of-distribution inputs over their non-Bayesian counterparts. Although, Deep ensemble methods (Seligmann et al. 2024; Lakshminarayanan, Pritzel, and Blundell 2017) have proven to be highly effective for Bayesian deep learning, their practical application is hindered by substantial computational cost. In this study, we introduce an innovative framework to mitigate the computational burden of ensemble Bayesian deep learning. We explore a more feasible alternative, inspired by the recent success of low-rank adapters, we introduce Bayesian Low-Rank LeArning (Bella). We show, i) Bella achieves a dramatic reduction in the number of trainable parameters required to approximate a Bayesian posterior; and ii) it not only maintains, but in some instances, surpasses the performance–in accuracy and out-of-distribution generalisation–of conventional Bayesian learning methods and non-Bayesian baselines. Our extensive empirical evaluation in large-scale tasks such as ImageNet, CAMELYON17, DomainNet, VQA with CLIP, LLaVA demonstrate the effectiveness and versatility of Bella in building highly scalable and practical Bayesian deep models for real-world applications.