Block Circulant Adapter for Large Language Models
Abstract
Fine-tuning large language models (LLMs) is difficult due to their huge model size. Recent Fourier domain-based methods show potential for reducing fine-tuning costs. We propose a block circulant matrix-based fine-tuning method with a stable training heuristic to leverage the properties of circulant matrices and one-dimensional Fourier transforms to reduce storage and computation costs. Experiments show that our method uses 14× less number of parameters than VeRA, 16× smaller than LoRA and 32× less FLOPs than FourierFT, while maintaining close or better task performance. Our approach presents a promising way in frequency domain to fine-tune large models on downstream tasks.
Cite
Text
Ding et al. "Block Circulant Adapter for Large Language Models." International Joint Conference on Artificial Intelligence, 2025. doi:10.24963/IJCAI.2025/560Markdown
[Ding et al. "Block Circulant Adapter for Large Language Models." International Joint Conference on Artificial Intelligence, 2025.](https://mlanthology.org/ijcai/2025/ding2025ijcai-block/) doi:10.24963/IJCAI.2025/560BibTeX
@inproceedings{ding2025ijcai-block,
title = {{Block Circulant Adapter for Large Language Models}},
author = {Ding, Xinyu and Wang, Meiqi and Liao, Siyu and Wang, Zhongfeng},
booktitle = {International Joint Conference on Artificial Intelligence},
year = {2025},
pages = {5030-5038},
doi = {10.24963/IJCAI.2025/560},
url = {https://mlanthology.org/ijcai/2025/ding2025ijcai-block/}
}