Private PoEtry: Private In-Context Learning via Product of Experts
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866918323750436864 |
|---|---|
| author | Romijnders, Rob Derakhshani, Mohammad Mahdi Petit, Jonathan Welling, Max Louizos, Christos Asano, Yuki M. |
| author_facet | Romijnders, Rob Derakhshani, Mohammad Mahdi Petit, Jonathan Welling, Max Louizos, Christos Asano, Yuki M. |
| contents | In-context learning (ICL) enables Large Language Models (LLMs) to adapt to new tasks with only a small set of examples at inference time, thereby avoiding task-specific fine-tuning. However, in-context examples may contain privacy-sensitive information that should not be revealed through model outputs. Existing differential privacy (DP) approaches to ICL are either computationally expensive or rely on heuristics with limited effectiveness, including context oversampling, synthetic data generation, or unnecessary thresholding. We reformulate private ICL through the lens of a Product-of-Experts model. This gives a theoretically grounded framework, and the algorithm can be trivially parallelized. We evaluate our method across five datasets in text classification, math, and vision-language. We find that our method improves accuracy by more than 30 percentage points on average compared to prior DP-ICL methods, while maintaining strong privacy guarantees. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_05012 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Private PoEtry: Private In-Context Learning via Product of Experts Romijnders, Rob Derakhshani, Mohammad Mahdi Petit, Jonathan Welling, Max Louizos, Christos Asano, Yuki M. Machine Learning In-context learning (ICL) enables Large Language Models (LLMs) to adapt to new tasks with only a small set of examples at inference time, thereby avoiding task-specific fine-tuning. However, in-context examples may contain privacy-sensitive information that should not be revealed through model outputs. Existing differential privacy (DP) approaches to ICL are either computationally expensive or rely on heuristics with limited effectiveness, including context oversampling, synthetic data generation, or unnecessary thresholding. We reformulate private ICL through the lens of a Product-of-Experts model. This gives a theoretically grounded framework, and the algorithm can be trivially parallelized. We evaluate our method across five datasets in text classification, math, and vision-language. We find that our method improves accuracy by more than 30 percentage points on average compared to prior DP-ICL methods, while maintaining strong privacy guarantees. |
| title | Private PoEtry: Private In-Context Learning via Product of Experts |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2602.05012 |