Guiding the Last Layer in Federated Learning with Pre-Trained Models

Abstract

Federated Learning (FL) is an emerging paradigm that enables a model to be trained across a number of participants without sharing data. While recent works have begun to consider the effects of using pre-trained models as an initialization point for existing FL algorithms, these approaches ignore the vast body of efficient transfer learning literature from the centralized learning setting. Here we revisit the problem of FL initialization from a pre-trained model considered in prior work and expand it to a set of computer vision transfer learning problems. We first show that simply fitting a linear classification head can be efficient and effective in many cases. Second we demonstrate that in the FL setting, fitting a classifier using the Nearest Class Means (NCM) can be done exactly and orders of magnitude more efficiently than existing proposals, while obtaining strong performance. Finally, we present that a two-phase approach of first obtaining the classifier and then fine-tuning the model can yield rapid convergence and improved generalization in the federated setting. We demonstrate the potential our method has to reduce communication and compute costs while achieving better model performance.