Improving Safety Alignment via Balanced Direct Preference Optimization
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917359381381120 |
|---|---|
| author | Zhao, Shiji Wang, Mengyang Xiong, Shukun Chen, Fangzhou Zhu, Qihui Ruan, Shouwei Xiao, Yisong Duan, Ranjie Chen, Xun Wei, XingXing |
| author_facet | Zhao, Shiji Wang, Mengyang Xiong, Shukun Chen, Fangzhou Zhu, Qihui Ruan, Shouwei Xiao, Yisong Duan, Ranjie Chen, Xun Wei, XingXing |
| contents | With the rapid development and widespread application of Large Language Models (LLMs), their potential safety risks have attracted widespread attention. Reinforcement Learning from Human Feedback (RLHF) has been adopted to enhance the safety performance of LLMs. As a simple and effective alternative to RLHF, Direct Preference Optimization (DPO) is widely used for safety alignment. However, safety alignment still suffers from severe overfitting, which limits its actual performance. This paper revisits the overfitting phenomenon from the perspective of the model's comprehension of the training data. We find that the Imbalanced Preference Comprehension phenomenon exists between responses in preference pairs, which compromises the model's safety performance. To address this, we propose Balanced Direct Preference Optimization (B-DPO), which adaptively modulates optimization strength between preferred and dispreferred responses based on mutual information. A series of experimental results show that B-DPO can enhance the safety capability while maintaining the competitive general capabilities of LLMs on various mainstream benchmarks compared to state-of-the-art methods. \color{red}{Warning: This paper contains examples of harmful texts, and reader discretion is recommended. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_22829 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Improving Safety Alignment via Balanced Direct Preference Optimization Zhao, Shiji Wang, Mengyang Xiong, Shukun Chen, Fangzhou Zhu, Qihui Ruan, Shouwei Xiao, Yisong Duan, Ranjie Chen, Xun Wei, XingXing Artificial Intelligence With the rapid development and widespread application of Large Language Models (LLMs), their potential safety risks have attracted widespread attention. Reinforcement Learning from Human Feedback (RLHF) has been adopted to enhance the safety performance of LLMs. As a simple and effective alternative to RLHF, Direct Preference Optimization (DPO) is widely used for safety alignment. However, safety alignment still suffers from severe overfitting, which limits its actual performance. This paper revisits the overfitting phenomenon from the perspective of the model's comprehension of the training data. We find that the Imbalanced Preference Comprehension phenomenon exists between responses in preference pairs, which compromises the model's safety performance. To address this, we propose Balanced Direct Preference Optimization (B-DPO), which adaptively modulates optimization strength between preferred and dispreferred responses based on mutual information. A series of experimental results show that B-DPO can enhance the safety capability while maintaining the competitive general capabilities of LLMs on various mainstream benchmarks compared to state-of-the-art methods. \color{red}{Warning: This paper contains examples of harmful texts, and reader discretion is recommended. |
| title | Improving Safety Alignment via Balanced Direct Preference Optimization |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2603.22829 |