Efficient Safety Alignment of Large Language Models via Preference Re-ranking and Representation-based Reward Modeling
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912431177990144 |
|---|---|
| author | Deng, Qiyuan Bai, Xuefeng Chen, Kehai Wang, Yaowei Nie, Liqiang Zhang, Min |
| author_facet | Deng, Qiyuan Bai, Xuefeng Chen, Kehai Wang, Yaowei Nie, Liqiang Zhang, Min |
| contents | Reinforcement Learning (RL) algorithms for safety alignment of Large Language Models (LLMs), such as Direct Preference Optimization (DPO), encounter the challenge of distribution shift. Current approaches typically address this issue through online sampling from the target policy, which requires significant computational resources. In this paper, we hypothesize that during off-policy training, while the ranking order of output generated by policy changes, their overall distribution remains relatively stable. This stability allows the conversion of the sampling process from the target policy into a computationally efficient re-ranking of preference data. Building on this hypothesis, we propose a new framework that leverages the model's intrinsic safety judgment capability to extract reward signals, which are then used to calculate label confidence for preference reordering. Extensive experiments and theoretical analysis demonstrate that the proposed method effectively addresses the distribution shift issue, remarkably enhancing the safety performance while avoiding about 300x computational overheads. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_10093 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Efficient Safety Alignment of Large Language Models via Preference Re-ranking and Representation-based Reward Modeling Deng, Qiyuan Bai, Xuefeng Chen, Kehai Wang, Yaowei Nie, Liqiang Zhang, Min Computation and Language Reinforcement Learning (RL) algorithms for safety alignment of Large Language Models (LLMs), such as Direct Preference Optimization (DPO), encounter the challenge of distribution shift. Current approaches typically address this issue through online sampling from the target policy, which requires significant computational resources. In this paper, we hypothesize that during off-policy training, while the ranking order of output generated by policy changes, their overall distribution remains relatively stable. This stability allows the conversion of the sampling process from the target policy into a computationally efficient re-ranking of preference data. Building on this hypothesis, we propose a new framework that leverages the model's intrinsic safety judgment capability to extract reward signals, which are then used to calculate label confidence for preference reordering. Extensive experiments and theoretical analysis demonstrate that the proposed method effectively addresses the distribution shift issue, remarkably enhancing the safety performance while avoiding about 300x computational overheads. |
| title | Efficient Safety Alignment of Large Language Models via Preference Re-ranking and Representation-based Reward Modeling |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2503.10093 |