Model Merging by Gradient Matching
Abstract
Models trained on different datasets can be merged by a weighted-averaging of their parameters, but why does it work and when can it fail? Here, we connect the inaccuracy of weighted-averaging to mismatches in the gradients and propose a new uncertainty-based scheme to improve the performance by reducing the mismatch. The connection also reveals implicit assumptions in other schemes such as averaging, task arithmetic, and Fisher-weighted averaging.
Cite
Text
Daheim et al. "Model Merging by Gradient Matching." NeurIPS 2023 Workshops: UniReps, 2023.Markdown
[Daheim et al. "Model Merging by Gradient Matching." NeurIPS 2023 Workshops: UniReps, 2023.](https://mlanthology.org/neuripsw/2023/daheim2023neuripsw-model/)BibTeX
@inproceedings{daheim2023neuripsw-model,
title = {{Model Merging by Gradient Matching}},
author = {Daheim, Nico and Möllenhoff, Thomas and Ponti, Edoardo and Gurevych, Iryna and Khan, Mohammad Emtiyaz},
booktitle = {NeurIPS 2023 Workshops: UniReps},
year = {2023},
url = {https://mlanthology.org/neuripsw/2023/daheim2023neuripsw-model/}
}