Scalable DP-SGD: Shuffling vs. Poisson Subsampling
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910687115083776 |
|---|---|
| author | Chua, Lynn Ghazi, Badih Kamath, Pritish Kumar, Ravi Manurangsi, Pasin Sinha, Amer Zhang, Chiyuan |
| author_facet | Chua, Lynn Ghazi, Badih Kamath, Pritish Kumar, Ravi Manurangsi, Pasin Sinha, Amer Zhang, Chiyuan |
| contents | We provide new lower bounds on the privacy guarantee of the multi-epoch Adaptive Batch Linear Queries (ABLQ) mechanism with shuffled batch sampling, demonstrating substantial gaps when compared to Poisson subsampling; prior analysis was limited to a single epoch. Since the privacy analysis of Differentially Private Stochastic Gradient Descent (DP-SGD) is obtained by analyzing the ABLQ mechanism, this brings into serious question the common practice of implementing shuffling-based DP-SGD, but reporting privacy parameters as if Poisson subsampling was used. To understand the impact of this gap on the utility of trained machine learning models, we introduce a practical approach to implement Poisson subsampling at scale using massively parallel computation, and efficiently train models with the same. We compare the utility of models trained with Poisson-subsampling-based DP-SGD, and the optimistic estimates of utility when using shuffling, via our new lower bounds on the privacy guarantee of ABLQ with shuffling. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_04205 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Scalable DP-SGD: Shuffling vs. Poisson Subsampling Chua, Lynn Ghazi, Badih Kamath, Pritish Kumar, Ravi Manurangsi, Pasin Sinha, Amer Zhang, Chiyuan Machine Learning Cryptography and Security Data Structures and Algorithms We provide new lower bounds on the privacy guarantee of the multi-epoch Adaptive Batch Linear Queries (ABLQ) mechanism with shuffled batch sampling, demonstrating substantial gaps when compared to Poisson subsampling; prior analysis was limited to a single epoch. Since the privacy analysis of Differentially Private Stochastic Gradient Descent (DP-SGD) is obtained by analyzing the ABLQ mechanism, this brings into serious question the common practice of implementing shuffling-based DP-SGD, but reporting privacy parameters as if Poisson subsampling was used. To understand the impact of this gap on the utility of trained machine learning models, we introduce a practical approach to implement Poisson subsampling at scale using massively parallel computation, and efficiently train models with the same. We compare the utility of models trained with Poisson-subsampling-based DP-SGD, and the optimistic estimates of utility when using shuffling, via our new lower bounds on the privacy guarantee of ABLQ with shuffling. |
| title | Scalable DP-SGD: Shuffling vs. Poisson Subsampling |
| topic | Machine Learning Cryptography and Security Data Structures and Algorithms |
| url | https://arxiv.org/abs/2411.04205 |