Bayesian Low-Rank Adaptation for Large Language Models
Abstract
Parameter-efficient fine-tuning (PEFT) has emerged as a new paradigm for cost-efficient fine-tuning of large language models (LLMs), with low-rank adaptation (LoRA) being a widely adopted choice. However, fine-tuned LLMs often become overconfident especially when fine-tuned on small datasets. Bayesian methods, with their inherent ability to estimate uncertainty, serve as potent tools to mitigate overconfidence and enhance calibration. In this work, we introduce Laplace-LoRA, a straightforward yet effective Bayesian method, which applies the Laplace approximation to the LoRA parameters and, considerably boosts the calibration of fine-tuned LLMs.
Cite
Text
Yang et al. "Bayesian Low-Rank Adaptation for Large Language Models." International Conference on Learning Representations, 2024.Markdown
[Yang et al. "Bayesian Low-Rank Adaptation for Large Language Models." International Conference on Learning Representations, 2024.](https://mlanthology.org/iclr/2024/yang2024iclr-bayesian/)BibTeX
@inproceedings{yang2024iclr-bayesian,
title = {{Bayesian Low-Rank Adaptation for Large Language Models}},
author = {Yang, Adam X. and Robeyns, Maxime and Wang, Xi and Aitchison, Laurence},
booktitle = {International Conference on Learning Representations},
year = {2024},
url = {https://mlanthology.org/iclr/2024/yang2024iclr-bayesian/}
}