AMoPO: Adaptive Multi-objective Preference Optimization without Reward Models and Reference Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912418950545408 |
|---|---|
| author | Liu, Qi Ruan, Jingqing Li, Hao Zhao, Haodong Wang, Desheng Chen, Jiansong Guanglu, Wan Cai, Xunliang Zheng, Zhi Xu, Tong |
| author_facet | Liu, Qi Ruan, Jingqing Li, Hao Zhao, Haodong Wang, Desheng Chen, Jiansong Guanglu, Wan Cai, Xunliang Zheng, Zhi Xu, Tong |
| contents | Existing multi-objective preference alignment methods for large language models (LLMs) face limitations: (1) the inability to effectively balance various preference dimensions, and (2) reliance on auxiliary reward/reference models introduces computational complexity. To address these challenges, we propose Adaptive Multi-objective Preference Optimization (AMoPO), a novel framework that achieves dynamic balance across preference dimensions. By introducing the multi-objective optimization paradigm to use the dimension-aware generation metrics as implicit rewards, AMoPO aligns LLMs with diverse preferences without additional reward models or reference models. We introduce an adaptive weight assignment mechanism that models the generation space as a Gaussian distribution, allowing dynamic prioritization of preference dimensions. Empirical results demonstrate that AMoPO outperforms state-of-the-art baselines by 28.5%, and the experiments on 7B, 14B, and 32B models reveal the scaling ability of AMoPO. Moreover, additional analysis of multiple dimensions verifies its adaptability and effectiveness. These findings validate AMoPO's capability to achieve dimension-aware preference alignment, highlighting its superiority. Our codes and datasets are available at https://github.com/Javkonline/AMoPO. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_07165 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | AMoPO: Adaptive Multi-objective Preference Optimization without Reward Models and Reference Models Liu, Qi Ruan, Jingqing Li, Hao Zhao, Haodong Wang, Desheng Chen, Jiansong Guanglu, Wan Cai, Xunliang Zheng, Zhi Xu, Tong Machine Learning Artificial Intelligence Existing multi-objective preference alignment methods for large language models (LLMs) face limitations: (1) the inability to effectively balance various preference dimensions, and (2) reliance on auxiliary reward/reference models introduces computational complexity. To address these challenges, we propose Adaptive Multi-objective Preference Optimization (AMoPO), a novel framework that achieves dynamic balance across preference dimensions. By introducing the multi-objective optimization paradigm to use the dimension-aware generation metrics as implicit rewards, AMoPO aligns LLMs with diverse preferences without additional reward models or reference models. We introduce an adaptive weight assignment mechanism that models the generation space as a Gaussian distribution, allowing dynamic prioritization of preference dimensions. Empirical results demonstrate that AMoPO outperforms state-of-the-art baselines by 28.5%, and the experiments on 7B, 14B, and 32B models reveal the scaling ability of AMoPO. Moreover, additional analysis of multiple dimensions verifies its adaptability and effectiveness. These findings validate AMoPO's capability to achieve dimension-aware preference alignment, highlighting its superiority. Our codes and datasets are available at https://github.com/Javkonline/AMoPO. |
| title | AMoPO: Adaptive Multi-objective Preference Optimization without Reward Models and Reference Models |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2506.07165 |