Omni-SMoLA: Boosting Generalist Multimodal Models with Soft Mixture of Low-Rank Experts
Abstract
In this work we present Omni-SMoLA a multimodal architecture that mixes many multi-modal experts efficiently and achieves both high specialist and generalist performance. In contrast to previous models for which we see performance degradation on average when training the models on a wide range of tasks we show that the SMoLA low-rank experts are able to model different skills and task and overall improve the performance of a generalist model. This finding indicates that simple LMM fine-tuning is suboptimal for handling a wide range of tasks and that pairing the act of fine-tuning with specifically-designed architecture changes leads to better performing models.
Cite
Text
Wu et al. "Omni-SMoLA: Boosting Generalist Multimodal Models with Soft Mixture of Low-Rank Experts." Conference on Computer Vision and Pattern Recognition, 2024. doi:10.1109/CVPR52733.2024.01347Markdown
[Wu et al. "Omni-SMoLA: Boosting Generalist Multimodal Models with Soft Mixture of Low-Rank Experts." Conference on Computer Vision and Pattern Recognition, 2024.](https://mlanthology.org/cvpr/2024/wu2024cvpr-omnismola/) doi:10.1109/CVPR52733.2024.01347BibTeX
@inproceedings{wu2024cvpr-omnismola,
title = {{Omni-SMoLA: Boosting Generalist Multimodal Models with Soft Mixture of Low-Rank Experts}},
author = {Wu, Jialin and Hu, Xia and Wang, Yaqing and Pang, Bo and Soricut, Radu},
booktitle = {Conference on Computer Vision and Pattern Recognition},
year = {2024},
pages = {14205-14215},
doi = {10.1109/CVPR52733.2024.01347},
url = {https://mlanthology.org/cvpr/2024/wu2024cvpr-omnismola/}
}