Choosing Public Datasets for Private Machine Learning via Gradient Subspace Distance
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915406905606144 |
|---|---|
| author | Gu, Xin Kamath, Gautam Wu, Zhiwei Steven |
| author_facet | Gu, Xin Kamath, Gautam Wu, Zhiwei Steven |
| contents | Differentially private stochastic gradient descent privatizes model training by injecting noise into each iteration, where the noise magnitude increases with the number of model parameters. Recent works suggest that we can reduce the noise by leveraging public data for private machine learning, by projecting gradients onto a subspace prescribed by the public data. However, given a choice of public datasets, it is not a priori clear which one may be most appropriate for the private task. We give an algorithm for selecting a public dataset by measuring a low-dimensional subspace distance between gradients of the public and private examples. We provide theoretical analysis demonstrating that the excess risk scales with this subspace distance. This distance is easy to compute and robust to modifications in the setting. Empirical evaluation shows that trained model accuracy is monotone in this distance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2303_01256 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Choosing Public Datasets for Private Machine Learning via Gradient Subspace Distance Gu, Xin Kamath, Gautam Wu, Zhiwei Steven Machine Learning Cryptography and Security Computer Vision and Pattern Recognition Data Structures and Algorithms Differentially private stochastic gradient descent privatizes model training by injecting noise into each iteration, where the noise magnitude increases with the number of model parameters. Recent works suggest that we can reduce the noise by leveraging public data for private machine learning, by projecting gradients onto a subspace prescribed by the public data. However, given a choice of public datasets, it is not a priori clear which one may be most appropriate for the private task. We give an algorithm for selecting a public dataset by measuring a low-dimensional subspace distance between gradients of the public and private examples. We provide theoretical analysis demonstrating that the excess risk scales with this subspace distance. This distance is easy to compute and robust to modifications in the setting. Empirical evaluation shows that trained model accuracy is monotone in this distance. |
| title | Choosing Public Datasets for Private Machine Learning via Gradient Subspace Distance |
| topic | Machine Learning Cryptography and Security Computer Vision and Pattern Recognition Data Structures and Algorithms |
| url | https://arxiv.org/abs/2303.01256 |