Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2510.07304 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911198738382848 |
|---|---|
| author | Kim, Donghwan Gu, Xin Baek, Jinho Lo, Timothy Min, Younghoon Shin, Kwangsik Kim, Jongryool Park, Jongse Maeng, Kiwan |
| author_facet | Kim, Donghwan Gu, Xin Baek, Jinho Lo, Timothy Min, Younghoon Shin, Kwangsik Kim, Jongryool Park, Jongse Maeng, Kiwan |
| contents | Machine learning (ML) models memorize and leak training data, causing serious privacy issues to data owners. Training algorithms with differential privacy (DP), such as DP-SGD, have been gaining attention as a solution. However, DP-SGD adds a noise at each training iteration, which degrades the accuracy of the trained model. To improve accuracy, a new family of approaches adds carefully designed correlated noises, so that noises cancel out each other across iterations. We performed an extensive characterization study of these new mechanisms, for the first time to the best of our knowledge, and show they incur non-negligible overheads when the model is large or uses large embedding tables. Motivated by the analysis, we propose Cocoon, a hardware-software co-designed framework for efficient training with correlated noises. Cocoon accelerates models with embedding tables through pre-computing and storing correlated noises in a coalesced format (Cocoon-Emb), and supports large models through a custom near-memory processing device (Cocoon-NMP). On a real system with an FPGA-based NMP device prototype, Cocoon improves the performance by 2.33-10.82x(Cocoon-Emb) and 1.55-3.06x (Cocoon-NMP). |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_07304 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Cocoon: A System Architecture for Differentially Private Training with Correlated Noises Kim, Donghwan Gu, Xin Baek, Jinho Lo, Timothy Min, Younghoon Shin, Kwangsik Kim, Jongryool Park, Jongse Maeng, Kiwan Hardware Architecture Artificial Intelligence Cryptography and Security Machine Learning Machine learning (ML) models memorize and leak training data, causing serious privacy issues to data owners. Training algorithms with differential privacy (DP), such as DP-SGD, have been gaining attention as a solution. However, DP-SGD adds a noise at each training iteration, which degrades the accuracy of the trained model. To improve accuracy, a new family of approaches adds carefully designed correlated noises, so that noises cancel out each other across iterations. We performed an extensive characterization study of these new mechanisms, for the first time to the best of our knowledge, and show they incur non-negligible overheads when the model is large or uses large embedding tables. Motivated by the analysis, we propose Cocoon, a hardware-software co-designed framework for efficient training with correlated noises. Cocoon accelerates models with embedding tables through pre-computing and storing correlated noises in a coalesced format (Cocoon-Emb), and supports large models through a custom near-memory processing device (Cocoon-NMP). On a real system with an FPGA-based NMP device prototype, Cocoon improves the performance by 2.33-10.82x(Cocoon-Emb) and 1.55-3.06x (Cocoon-NMP). |
| title | Cocoon: A System Architecture for Differentially Private Training with Correlated Noises |
| topic | Hardware Architecture Artificial Intelligence Cryptography and Security Machine Learning |
| url | https://arxiv.org/abs/2510.07304 |