Scalable Continual Learning: Adaptive MoEs for Expanding Task Sets
Abstract
Recently, the Mixture-of-Experts (MoE) model has been shown to be an effective strategy for continual learning because it can adapt to a range of tasks by employing an array of "experts'' that each specialize on certain tasks. However, the MoE model lacks the ability to adapt to completely new tasks, particularly as the number of tasks grows to be large. In this work we develop a framework for expanding the number of experts as needed when new tasks arise. We also provide simulations demonstrating that our approach can effectively handle a growing number of tasks.
Cite
Text
Candocia et al. "Scalable Continual Learning: Adaptive MoEs for Expanding Task Sets." ICLR 2025 Workshops: SLLM, 2025.Markdown
[Candocia et al. "Scalable Continual Learning: Adaptive MoEs for Expanding Task Sets." ICLR 2025 Workshops: SLLM, 2025.](https://mlanthology.org/iclrw/2025/candocia2025iclrw-scalable/)BibTeX
@inproceedings{candocia2025iclrw-scalable,
title = {{Scalable Continual Learning: Adaptive MoEs for Expanding Task Sets}},
author = {Candocia, Adrian and Inan, Omer Mustafa and Agarwal, Raaghav and Varma, Aamod and Davenport, Mark A.},
booktitle = {ICLR 2025 Workshops: SLLM},
year = {2025},
url = {https://mlanthology.org/iclrw/2025/candocia2025iclrw-scalable/}
}