Adaptive Helpfulness-Harmlessness Alignment with Preference Vectors
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866908810356981760 |
|---|---|
| author | Liang, Ren-Wei Hsu, Chin-Ting Yu, Chan-Hung Agrawal, Saransh Huang, Shih-Cheng Lin, Chieh-Yen Chen, Shang-Tse Huang, Kuan-Hao Sun, Shao-Hua |
| author_facet | Liang, Ren-Wei Hsu, Chin-Ting Yu, Chan-Hung Agrawal, Saransh Huang, Shih-Cheng Lin, Chieh-Yen Chen, Shang-Tse Huang, Kuan-Hao Sun, Shao-Hua |
| contents | Ensuring that large language models (LLMs) are both helpful and harmless is a critical challenge, as overly strict constraints can lead to excessive refusals, while permissive models risk generating harmful content. Existing approaches, such as reinforcement learning from human feedback (RLHF) and direct preference optimization (DPO), attempt to balance these trade-offs but suffer from performance conflicts, limited controllability, and poor extendability. To address these issues, we propose Preference Vector, a novel framework inspired by task arithmetic. Instead of optimizing multiple preferences within a single objective, we train separate models on individual preferences, extract behavior shifts as preference vectors, and dynamically merge them at test time. This modular approach enables fine-grained, user-controllable preference adjustments and facilitates seamless integration of new preferences without retraining. Experiments show that our proposed Preference Vector framework improves helpfulness without excessive conservatism, allows smooth control over preference trade-offs, and supports scalable multi-preference alignment. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_20106 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Adaptive Helpfulness-Harmlessness Alignment with Preference Vectors Liang, Ren-Wei Hsu, Chin-Ting Yu, Chan-Hung Agrawal, Saransh Huang, Shih-Cheng Lin, Chieh-Yen Chen, Shang-Tse Huang, Kuan-Hao Sun, Shao-Hua Machine Learning Artificial Intelligence Ensuring that large language models (LLMs) are both helpful and harmless is a critical challenge, as overly strict constraints can lead to excessive refusals, while permissive models risk generating harmful content. Existing approaches, such as reinforcement learning from human feedback (RLHF) and direct preference optimization (DPO), attempt to balance these trade-offs but suffer from performance conflicts, limited controllability, and poor extendability. To address these issues, we propose Preference Vector, a novel framework inspired by task arithmetic. Instead of optimizing multiple preferences within a single objective, we train separate models on individual preferences, extract behavior shifts as preference vectors, and dynamically merge them at test time. This modular approach enables fine-grained, user-controllable preference adjustments and facilitates seamless integration of new preferences without retraining. Experiments show that our proposed Preference Vector framework improves helpfulness without excessive conservatism, allows smooth control over preference trade-offs, and supports scalable multi-preference alignment. |
| title | Adaptive Helpfulness-Harmlessness Alignment with Preference Vectors |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2504.20106 |