DP-LAC: Lightweight Adaptive Clipping for Differentially Private Federated Fine-tuning of Language Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916000249675776 |
|---|---|
| author | Mehmood, Haaris Xu, Jie Saravanan, Karthikeyan Van Dalen, Rogier Ozay, Mete |
| author_facet | Mehmood, Haaris Xu, Jie Saravanan, Karthikeyan Van Dalen, Rogier Ozay, Mete |
| contents | Federated learning (FL) enables the collaborative training of large-scale language models (LLMs) across edge devices while keeping user data on-device. However, FL still exposes sensitive information through client-provided gradients. Differentially private stochastic gradient descent (DP-SGD) mitigates this risk by clipping each client's contribution to a threshold $C$ and adding noise proportional to $C$. Existing adaptive clipping techniques dynamically adjust $C$ but demand tedious hyperparameter tuning, which can erode the privacy budget. In this paper, we introduce DP-LAC, a method that first estimates an initial clipping threshold within an order of magnitude of the optimum using private histogram estimation, and then adapts this threshold during training without consuming additional privacy budget or introducing new hyperparameters. Empirical results show that DP-LAC outperforms both state-of-the-art adaptive clipping methods and vanilla DP-SGD, achieving an average accuracy gain of $6.6\%$. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_10272 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | DP-LAC: Lightweight Adaptive Clipping for Differentially Private Federated Fine-tuning of Language Models Mehmood, Haaris Xu, Jie Saravanan, Karthikeyan Van Dalen, Rogier Ozay, Mete Machine Learning Artificial Intelligence Cryptography and Security Distributed, Parallel, and Cluster Computing Federated learning (FL) enables the collaborative training of large-scale language models (LLMs) across edge devices while keeping user data on-device. However, FL still exposes sensitive information through client-provided gradients. Differentially private stochastic gradient descent (DP-SGD) mitigates this risk by clipping each client's contribution to a threshold $C$ and adding noise proportional to $C$. Existing adaptive clipping techniques dynamically adjust $C$ but demand tedious hyperparameter tuning, which can erode the privacy budget. In this paper, we introduce DP-LAC, a method that first estimates an initial clipping threshold within an order of magnitude of the optimum using private histogram estimation, and then adapts this threshold during training without consuming additional privacy budget or introducing new hyperparameters. Empirical results show that DP-LAC outperforms both state-of-the-art adaptive clipping methods and vanilla DP-SGD, achieving an average accuracy gain of $6.6\%$. |
| title | DP-LAC: Lightweight Adaptive Clipping for Differentially Private Federated Fine-tuning of Language Models |
| topic | Machine Learning Artificial Intelligence Cryptography and Security Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2605.10272 |