Adversarial Preference Learning for Robust LLM Alignment
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913866952212480 |
|---|---|
| author | Wang, Yuanfu Wang, Pengyu Xi, Chenyang Tang, Bo Zhu, Junyi Wei, Wenqiang Chen, Chen Yang, Chao Zhang, Jingfeng Lu, Chaochao Niu, Yijun Mao, Keming Li, Zhiyu Xiong, Feiyu Hu, Jie Yang, Mingchuan |
| author_facet | Wang, Yuanfu Wang, Pengyu Xi, Chenyang Tang, Bo Zhu, Junyi Wei, Wenqiang Chen, Chen Yang, Chao Zhang, Jingfeng Lu, Chaochao Niu, Yijun Mao, Keming Li, Zhiyu Xiong, Feiyu Hu, Jie Yang, Mingchuan |
| contents | Modern language models often rely on Reinforcement Learning from Human Feedback (RLHF) to encourage safe behaviors. However, they remain vulnerable to adversarial attacks due to three key limitations: (1) the inefficiency and high cost of human annotation, (2) the vast diversity of potential adversarial attacks, and (3) the risk of feedback bias and reward hacking. To address these challenges, we introduce Adversarial Preference Learning (APL), an iterative adversarial training method incorporating three key innovations. First, a direct harmfulness metric based on the model's intrinsic preference probabilities, eliminating reliance on external assessment. Second, a conditional generative attacker that synthesizes input-specific adversarial variations. Third, an iterative framework with automated closed-loop feedback, enabling continuous adaptation through vulnerability discovery and mitigation. Experiments on Mistral-7B-Instruct-v0.3 demonstrate that APL significantly enhances robustness, achieving 83.33% harmlessness win rate over the base model (evaluated by GPT-4o), reducing harmful outputs from 5.88% to 0.43% (measured by LLaMA-Guard), and lowering attack success rate by up to 65% according to HarmBench. Notably, APL maintains competitive utility, with an MT-Bench score of 6.59 (comparable to the baseline 6.78) and an LC-WinRate of 46.52% against the base model. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_24369 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Adversarial Preference Learning for Robust LLM Alignment Wang, Yuanfu Wang, Pengyu Xi, Chenyang Tang, Bo Zhu, Junyi Wei, Wenqiang Chen, Chen Yang, Chao Zhang, Jingfeng Lu, Chaochao Niu, Yijun Mao, Keming Li, Zhiyu Xiong, Feiyu Hu, Jie Yang, Mingchuan Machine Learning Artificial Intelligence Modern language models often rely on Reinforcement Learning from Human Feedback (RLHF) to encourage safe behaviors. However, they remain vulnerable to adversarial attacks due to three key limitations: (1) the inefficiency and high cost of human annotation, (2) the vast diversity of potential adversarial attacks, and (3) the risk of feedback bias and reward hacking. To address these challenges, we introduce Adversarial Preference Learning (APL), an iterative adversarial training method incorporating three key innovations. First, a direct harmfulness metric based on the model's intrinsic preference probabilities, eliminating reliance on external assessment. Second, a conditional generative attacker that synthesizes input-specific adversarial variations. Third, an iterative framework with automated closed-loop feedback, enabling continuous adaptation through vulnerability discovery and mitigation. Experiments on Mistral-7B-Instruct-v0.3 demonstrate that APL significantly enhances robustness, achieving 83.33% harmlessness win rate over the base model (evaluated by GPT-4o), reducing harmful outputs from 5.88% to 0.43% (measured by LLaMA-Guard), and lowering attack success rate by up to 65% according to HarmBench. Notably, APL maintains competitive utility, with an MT-Bench score of 6.59 (comparable to the baseline 6.78) and an LC-WinRate of 46.52% against the base model. |
| title | Adversarial Preference Learning for Robust LLM Alignment |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2505.24369 |