FediOS: Decoupling Orthogonal Subspaces for Personalization in Feature-Skew Federated Learning

Abstract

Personalized federated learning (pFL) enables collaborative training among multiple clients to enhance the capability of customized local models. In pFL, clients may have heterogeneous (also known as non-IID) data, which poses a key challenge in how to decouple the data knowledge into generic knowledge for global sharing and personalized knowledge for preserving local personalization. A typical way of pFL focuses on label distribution skew, and they adopt a decoupling scheme where the model is split into a common feature extractor and two prediction heads (generic and personalized). However, such a decoupling scheme cannot solve the essential problem of feature-skew heterogeneity, because one feature extractor only outputs one feature map that may not realize generic and personalized features at the same time. Therefore, in this paper, we rethink the decoupling in feature-skew pFL and subsequently propose FediOS inspired by orthogonal techniques in continual learning. In FediOS, we decouple the features by implementing two feature extractors (generic and personalized) and realize implicit feature decoupling by set-and-fixed orthogonal projections. For two feature extractors, orthogonal projections are used to map the generic features into one common subspace and scatter the personalized features into different subspaces. The proposed shared prediction head can adapt to the sample-wise importance of generic and personalized features for prediction, given that samples would have distinct proportions of generic and personalized features. Extensive experiments on four vision datasets demonstrate our method reaches state-of-the-art pFL performances under feature skew, even label skew, and mix-heterogeneity of both label and feature skew.