DP-Adam: Correcting DP Bias in Adam's Second Moment Estimation
Abstract
We observe that the traditional use of DP with the Adam optimizer introduces a bias in the second moment estimation, due to the addition of independent noise in the gradient computation. This bias leads to a different scaling for low variance parameter updates, that is inconsistent with the behavior of non-private Adam, and Adam's sign descent interpretation. Empirically, correcting the bias introduced by DP noise significantly improves the optimization performance of DP-Adam.
Cite
Text
Tang and Lécuyer. "DP-Adam: Correcting DP Bias in Adam's Second Moment Estimation." ICLR 2023 Workshops: RTML, 2023.Markdown
[Tang and Lécuyer. "DP-Adam: Correcting DP Bias in Adam's Second Moment Estimation." ICLR 2023 Workshops: RTML, 2023.](https://mlanthology.org/iclrw/2023/tang2023iclrw-dpadam/)BibTeX
@inproceedings{tang2023iclrw-dpadam,
title = {{DP-Adam: Correcting DP Bias in Adam's Second Moment Estimation}},
author = {Tang, Qiaoyue and Lécuyer, Mathias},
booktitle = {ICLR 2023 Workshops: RTML},
year = {2023},
url = {https://mlanthology.org/iclrw/2023/tang2023iclrw-dpadam/}
}