Nearly Optimal Bounds for Cyclic Forgetting
Abstract
We provide theoretical bounds on the forgetting quantity in the continual learning setting for linear tasks, where each round of learning corresponds to projecting onto a linear subspace. For a cyclic task ordering on $T$ tasks repeated $m$ times each, we prove the best known upper bound of $O(T^2/m)$ on the forgetting. Notably, our bound holds uniformly over all choices of tasks and is independent of the ambient dimension. Our main technical contribution is a characterization of the union of all numerical ranges of products of $T$ (real or complex) projections as a sinusoidal spiral, which may be of independent interest.
Cite
Text
Swartworth et al. "Nearly Optimal Bounds for Cyclic Forgetting." Neural Information Processing Systems, 2023.Markdown
[Swartworth et al. "Nearly Optimal Bounds for Cyclic Forgetting." Neural Information Processing Systems, 2023.](https://mlanthology.org/neurips/2023/swartworth2023neurips-nearly/)BibTeX
@inproceedings{swartworth2023neurips-nearly,
title = {{Nearly Optimal Bounds for Cyclic Forgetting}},
author = {Swartworth, William and Needell, Deanna and Ward, Rachel and Kong, Mark and Jeong, Halyun},
booktitle = {Neural Information Processing Systems},
year = {2023},
url = {https://mlanthology.org/neurips/2023/swartworth2023neurips-nearly/}
}