Correlating Cross-Iteration Noise for DP-SGD using Model Curvature
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911533255098368 |
|---|---|
| author | Gu, Xin Xiao, Yingtai He, Guanlin Bai, Jiamu Kifer, Daniel Maeng, Kiwan |
| author_facet | Gu, Xin Xiao, Yingtai He, Guanlin Bai, Jiamu Kifer, Daniel Maeng, Kiwan |
| contents | Differentially private stochastic gradient descent (DP-SGD) offers the promise of training deep learning models while mitigating many privacy risks. However, there is currently a large accuracy gap between DP-SGD and normal SGD training. This has resulted in different lines of research investigating orthogonal ways of improving privacy-preserving training. One such line of work, known as DP-MF, correlates the privacy noise across different iterations of stochastic gradient descent -- allowing later iterations to cancel out some of the noise added to earlier iterations. In this paper, we study how to improve this noise correlation. We propose a technique called NoiseCurve that uses model curvature, estimated from public unlabeled data, to improve the quality of this cross-iteration noise correlation. Our experiments on various datasets, models, and privacy parameters show that the noise correlations computed by NoiseCurve offer consistent and significant improvements in accuracy over the correlation scheme used by DP-MF. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_05416 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Correlating Cross-Iteration Noise for DP-SGD using Model Curvature Gu, Xin Xiao, Yingtai He, Guanlin Bai, Jiamu Kifer, Daniel Maeng, Kiwan Machine Learning Differentially private stochastic gradient descent (DP-SGD) offers the promise of training deep learning models while mitigating many privacy risks. However, there is currently a large accuracy gap between DP-SGD and normal SGD training. This has resulted in different lines of research investigating orthogonal ways of improving privacy-preserving training. One such line of work, known as DP-MF, correlates the privacy noise across different iterations of stochastic gradient descent -- allowing later iterations to cancel out some of the noise added to earlier iterations. In this paper, we study how to improve this noise correlation. We propose a technique called NoiseCurve that uses model curvature, estimated from public unlabeled data, to improve the quality of this cross-iteration noise correlation. Our experiments on various datasets, models, and privacy parameters show that the noise correlations computed by NoiseCurve offer consistent and significant improvements in accuracy over the correlation scheme used by DP-MF. |
| title | Correlating Cross-Iteration Noise for DP-SGD using Model Curvature |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2510.05416 |