Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2407.06443 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916708158013440 |
|---|---|
| author | Feng, Qizhang Kasa, Siva Rajesh Kasa, Santhosh Kumar Yun, Hyokun Teo, Choon Hui Bodapati, Sravan Babu |
| author_facet | Feng, Qizhang Kasa, Siva Rajesh Kasa, Santhosh Kumar Yun, Hyokun Teo, Choon Hui Bodapati, Sravan Babu |
| contents | Large Language Models (LLMs) have seen widespread adoption due to their remarkable natural language capabilities. However, when deploying them in real-world settings, it is important to align LLMs to generate texts according to acceptable human standards. Methods such as Proximal Policy Optimization (PPO) and Direct Preference Optimization (DPO) have enabled significant progress in refining LLMs using human preference data. However, the privacy concerns inherent in utilizing such preference data have yet to be adequately studied. In this paper, we investigate the vulnerability of LLMs aligned using two widely used methods - DPO and PPO - to membership inference attacks (MIAs). Our study has two main contributions: first, we theoretically motivate that DPO models are more vulnerable to MIA compared to PPO models; second, we introduce a novel reference-based attack framework specifically for analyzing preference data called PREMIA (\uline{Pre}ference data \uline{MIA}). Using PREMIA and existing baselines we empirically show that DPO models have a relatively heightened vulnerability towards MIA. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_06443 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Exposing Privacy Gaps: Membership Inference Attack on Preference Data for LLM Alignment Feng, Qizhang Kasa, Siva Rajesh Kasa, Santhosh Kumar Yun, Hyokun Teo, Choon Hui Bodapati, Sravan Babu Artificial Intelligence Large Language Models (LLMs) have seen widespread adoption due to their remarkable natural language capabilities. However, when deploying them in real-world settings, it is important to align LLMs to generate texts according to acceptable human standards. Methods such as Proximal Policy Optimization (PPO) and Direct Preference Optimization (DPO) have enabled significant progress in refining LLMs using human preference data. However, the privacy concerns inherent in utilizing such preference data have yet to be adequately studied. In this paper, we investigate the vulnerability of LLMs aligned using two widely used methods - DPO and PPO - to membership inference attacks (MIAs). Our study has two main contributions: first, we theoretically motivate that DPO models are more vulnerable to MIA compared to PPO models; second, we introduce a novel reference-based attack framework specifically for analyzing preference data called PREMIA (\uline{Pre}ference data \uline{MIA}). Using PREMIA and existing baselines we empirically show that DPO models have a relatively heightened vulnerability towards MIA. |
| title | Exposing Privacy Gaps: Membership Inference Attack on Preference Data for LLM Alignment |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2407.06443 |