LazyDP: Co-Designing Algorithm-Software for Scalable Training of Differentially Private Recommendation Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914753481277440 |
|---|---|
| author | Lim, Juntaek Kwon, Youngeun Hwang, Ranggi Maeng, Kiwan Suh, G. Edward Rhu, Minsoo |
| author_facet | Lim, Juntaek Kwon, Youngeun Hwang, Ranggi Maeng, Kiwan Suh, G. Edward Rhu, Minsoo |
| contents | Differential privacy (DP) is widely being employed in the industry as a practical standard for privacy protection. While private training of computer vision or natural language processing applications has been studied extensively, the computational challenges of training of recommender systems (RecSys) with DP have not been explored. In this work, we first present our detailed characterization of private RecSys training using DP-SGD, root-causing its several performance bottlenecks. Specifically, we identify DP-SGD's noise sampling and noisy gradient update stage to suffer from a severe compute and memory bandwidth limitation, respectively, causing significant performance overhead in training private RecSys. Based on these findings, we propose LazyDP, an algorithm-software co-design that addresses the compute and memory challenges of training RecSys with DP-SGD. Compared to a state-of-the-art DP-SGD training system, we demonstrate that LazyDP provides an average 119x training throughput improvement while also ensuring mathematically equivalent, differentially private RecSys models to be trained. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_08847 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LazyDP: Co-Designing Algorithm-Software for Scalable Training of Differentially Private Recommendation Models Lim, Juntaek Kwon, Youngeun Hwang, Ranggi Maeng, Kiwan Suh, G. Edward Rhu, Minsoo Information Retrieval Cryptography and Security Machine Learning Differential privacy (DP) is widely being employed in the industry as a practical standard for privacy protection. While private training of computer vision or natural language processing applications has been studied extensively, the computational challenges of training of recommender systems (RecSys) with DP have not been explored. In this work, we first present our detailed characterization of private RecSys training using DP-SGD, root-causing its several performance bottlenecks. Specifically, we identify DP-SGD's noise sampling and noisy gradient update stage to suffer from a severe compute and memory bandwidth limitation, respectively, causing significant performance overhead in training private RecSys. Based on these findings, we propose LazyDP, an algorithm-software co-design that addresses the compute and memory challenges of training RecSys with DP-SGD. Compared to a state-of-the-art DP-SGD training system, we demonstrate that LazyDP provides an average 119x training throughput improvement while also ensuring mathematically equivalent, differentially private RecSys models to be trained. |
| title | LazyDP: Co-Designing Algorithm-Software for Scalable Training of Differentially Private Recommendation Models |
| topic | Information Retrieval Cryptography and Security Machine Learning |
| url | https://arxiv.org/abs/2404.08847 |