Federated Learning with Position-Aware Neurons
Abstract
Federated Learning (FL) fuses collaborative models from local nodes without centralizing users' data. The permutation invariance property of neural networks and the non-i.i.d. data across clients make the locally updated parameters imprecisely aligned, disabling the coordinate-based parameter averaging. Traditional neurons do not explicitly consider position information. Hence, we propose Position-Aware Neurons (PANs) as an alternative, fusing position-related values (i.e., position encodings) into neuron outputs. PANs couple themselves to their positions and minimize the possibility of dislocation, even updating on heterogeneous data. We turn on/off PANs to disable/enable the permutation invariance property of neural networks. PANs are tightly coupled with positions when applied to FL, making parameters across clients pre-aligned and facilitating coordinate-based parameter averaging. PANs are algorithm-agnostic and could universally improve existing FL algorithms. Furthermore, "FL with PANs" is simple to implement and computationally friendly.
Cite
Text
Li et al. "Federated Learning with Position-Aware Neurons." Conference on Computer Vision and Pattern Recognition, 2022. doi:10.1109/CVPR52688.2022.00984Markdown
[Li et al. "Federated Learning with Position-Aware Neurons." Conference on Computer Vision and Pattern Recognition, 2022.](https://mlanthology.org/cvpr/2022/li2022cvpr-federated/) doi:10.1109/CVPR52688.2022.00984BibTeX
@inproceedings{li2022cvpr-federated,
title = {{Federated Learning with Position-Aware Neurons}},
author = {Li, Xin-Chun and Xu, Yi-Chu and Song, Shaoming and Li, Bingshuai and Li, Yinchuan and Shao, Yunfeng and Zhan, De-Chuan},
booktitle = {Conference on Computer Vision and Pattern Recognition},
year = {2022},
pages = {10082-10091},
doi = {10.1109/CVPR52688.2022.00984},
url = {https://mlanthology.org/cvpr/2022/li2022cvpr-federated/}
}