Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2501.18914 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909471067865088 |
|---|---|
| author | McKenna, Ryan Huang, Yangsibo Sinha, Amer Balle, Borja Charles, Zachary Choquette-Choo, Christopher A. Ghazi, Badih Kaissis, George Kumar, Ravi Liu, Ruibo Yu, Da Zhang, Chiyuan |
| author_facet | McKenna, Ryan Huang, Yangsibo Sinha, Amer Balle, Borja Charles, Zachary Choquette-Choo, Christopher A. Ghazi, Badih Kaissis, George Kumar, Ravi Liu, Ruibo Yu, Da Zhang, Chiyuan |
| contents | Scaling laws have emerged as important components of large language model (LLM) training as they can predict performance gains through scale, and provide guidance on important hyper-parameter choices that would otherwise be expensive. LLMs also rely on large, high-quality training datasets, like those sourced from (sometimes sensitive) user data. Training models on this sensitive user data requires careful privacy protections like differential privacy (DP). However, the dynamics of DP training are significantly different, and consequently their scaling laws are not yet fully understood. In this work, we establish scaling laws that accurately model the intricacies of DP LLM training, providing a complete picture of the compute-privacy-utility tradeoffs and the optimal training configurations in many settings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_18914 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Scaling Laws for Differentially Private Language Models McKenna, Ryan Huang, Yangsibo Sinha, Amer Balle, Borja Charles, Zachary Choquette-Choo, Christopher A. Ghazi, Badih Kaissis, George Kumar, Ravi Liu, Ruibo Yu, Da Zhang, Chiyuan Machine Learning Cryptography and Security Scaling laws have emerged as important components of large language model (LLM) training as they can predict performance gains through scale, and provide guidance on important hyper-parameter choices that would otherwise be expensive. LLMs also rely on large, high-quality training datasets, like those sourced from (sometimes sensitive) user data. Training models on this sensitive user data requires careful privacy protections like differential privacy (DP). However, the dynamics of DP training are significantly different, and consequently their scaling laws are not yet fully understood. In this work, we establish scaling laws that accurately model the intricacies of DP LLM training, providing a complete picture of the compute-privacy-utility tradeoffs and the optimal training configurations in many settings. |
| title | Scaling Laws for Differentially Private Language Models |
| topic | Machine Learning Cryptography and Security |
| url | https://arxiv.org/abs/2501.18914 |