PARS-Push: Personalized, Asynchronous and Robust Decentralized Optimization

Abstract

We study the decentralized multi-step Model-Agnostic Meta-Learning (MAML) framework where a group of $n$ agents seeks to find a common point that enables ``few-shot'' learning (personalization) via local stochastic gradient steps on their local functions. We formulate the personalized optimization problem under the MAML framework and propose PARS-Push, a decentralized asynchronous algorithm robust to message failures, communication delays, and directed message sharing. We characterize the convergence rate of PARS-Push for smooth and strongly convex and smooth and non-convex functions under arbitrary multi-step personalization. Moreover, we provide numerical experiments showing its performance under heterogeneous data setups.