NoEsis: Differentially Private Knowledge Transfer in Modular LLM Adaptation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909594038566912 |
|---|---|
| author | Romijnders, Rob Laskaridis, Stefanos Shamsabadi, Ali Shahin Haddadi, Hamed |
| author_facet | Romijnders, Rob Laskaridis, Stefanos Shamsabadi, Ali Shahin Haddadi, Hamed |
| contents | Large Language Models (LLM) are typically trained on vast amounts of data from various sources. Even when designed modularly (e.g., Mixture-of-Experts), LLMs can leak privacy on their sources. Conversely, training such models in isolation arguably prohibits generalization. To this end, we propose a framework, NoEsis, which builds upon the desired properties of modularity, privacy, and knowledge transfer. NoEsis integrates differential privacy with a hybrid two-staged parameter-efficient fine-tuning that combines domain-specific low-rank adapters, acting as experts, with common prompt tokens, acting as a knowledge-sharing backbone. Results from our evaluation on CodeXGLUE showcase that NoEsis can achieve provable privacy guarantees with tangible knowledge transfer across domains, and empirically show protection against Membership Inference Attacks. Finally, on code completion tasks, NoEsis bridges at least 77% of the accuracy gap between the non-shared and the non-private baseline. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_18147 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | NoEsis: Differentially Private Knowledge Transfer in Modular LLM Adaptation Romijnders, Rob Laskaridis, Stefanos Shamsabadi, Ali Shahin Haddadi, Hamed Cryptography and Security Machine Learning Large Language Models (LLM) are typically trained on vast amounts of data from various sources. Even when designed modularly (e.g., Mixture-of-Experts), LLMs can leak privacy on their sources. Conversely, training such models in isolation arguably prohibits generalization. To this end, we propose a framework, NoEsis, which builds upon the desired properties of modularity, privacy, and knowledge transfer. NoEsis integrates differential privacy with a hybrid two-staged parameter-efficient fine-tuning that combines domain-specific low-rank adapters, acting as experts, with common prompt tokens, acting as a knowledge-sharing backbone. Results from our evaluation on CodeXGLUE showcase that NoEsis can achieve provable privacy guarantees with tangible knowledge transfer across domains, and empirically show protection against Membership Inference Attacks. Finally, on code completion tasks, NoEsis bridges at least 77% of the accuracy gap between the non-shared and the non-private baseline. |
| title | NoEsis: Differentially Private Knowledge Transfer in Modular LLM Adaptation |
| topic | Cryptography and Security Machine Learning |
| url | https://arxiv.org/abs/2504.18147 |