Synergies Between Disentanglement and Sparsity: Generalization and Identifiability in Multi-Task Learning
Abstract
Although disentangled representations are often said to be beneficial for downstream tasks, current empirical and theoretical understanding is limited. In this work, we provide evidence that disentangled representations coupled with sparse task-specific predictors improve generalization. In the context of multi-task learning, we prove a new identifiability result that provides conditions under which maximally sparse predictors yield disentangled representations. Motivated by this theoretical result, we propose a practical approach to learn disentangled representations based on a sparsity-promoting bi-level optimization problem. Finally, we explore a meta-learning version of this algorithm based on group Lasso multiclass SVM predictors, for which we derive a tractable dual formulation. It obtains competitive results on standard few-shot classification benchmarks, while each task is using only a fraction of the learned representations.
Cite
Text
Lachapelle et al. "Synergies Between Disentanglement and Sparsity: Generalization and Identifiability in Multi-Task Learning." International Conference on Machine Learning, 2023.Markdown
[Lachapelle et al. "Synergies Between Disentanglement and Sparsity: Generalization and Identifiability in Multi-Task Learning." International Conference on Machine Learning, 2023.](https://mlanthology.org/icml/2023/lachapelle2023icml-synergies/)BibTeX
@inproceedings{lachapelle2023icml-synergies,
title = {{Synergies Between Disentanglement and Sparsity: Generalization and Identifiability in Multi-Task Learning}},
author = {Lachapelle, Sebastien and Deleu, Tristan and Mahajan, Divyat and Mitliagkas, Ioannis and Bengio, Yoshua and Lacoste-Julien, Simon and Bertrand, Quentin},
booktitle = {International Conference on Machine Learning},
year = {2023},
pages = {18171-18206},
volume = {202},
url = {https://mlanthology.org/icml/2023/lachapelle2023icml-synergies/}
}