MaPPO: Maximum a Posteriori Preference Optimization with Prior Knowledge
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910199079501824 |
|---|---|
| author | Lan, Guangchen Zhang, Sipeng Wang, Tianle Zhang, Yuwei Zhang, Daoan Wei, Xinpeng Pan, Xiaoman Zhang, Hongming Han, Dong-Jun Brinton, Christopher G. |
| author_facet | Lan, Guangchen Zhang, Sipeng Wang, Tianle Zhang, Yuwei Zhang, Daoan Wei, Xinpeng Pan, Xiaoman Zhang, Hongming Han, Dong-Jun Brinton, Christopher G. |
| contents | As the era of large language models (LLMs) unfolds, Preference Optimization (PO) methods have become a central approach to aligning LLMs with human preferences and improving performance. We propose Maximum a Posteriori Preference Optimization (MaPPO), a methodology for learning from preferences that explicitly incorporates prior reward knowledge into the optimization objective. Building on the paradigm employed by Direct Preference Optimization (DPO) and its variants of treating preference learning as a Maximum Likelihood Estimation (MLE) problem, MaPPO integrates prior reward estimates into a principled Maximum a Posteriori (MaP) objective. This not only generalizes DPO and its variants, but also enhances alignment by mitigating the oversimplified binary classification of responses. Additionally, MaPPO introduces no additional hyperparameters, and supports preference optimization in both offline and online settings. In addition, MaPPO can be used as a plugin for DPO variants, including widely used SimPO, IPO and CPO, and produce consistent improvements. Extensive empirical evaluations of different model sizes and model series on three standard benchmarks (MT-Bench, AlpacaEval 2.0, and Arena-Hard) demonstrate consistent improvements in alignment performance without sacrificing computational efficiency. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_21183 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MaPPO: Maximum a Posteriori Preference Optimization with Prior Knowledge Lan, Guangchen Zhang, Sipeng Wang, Tianle Zhang, Yuwei Zhang, Daoan Wei, Xinpeng Pan, Xiaoman Zhang, Hongming Han, Dong-Jun Brinton, Christopher G. Machine Learning Artificial Intelligence Computation and Language I.2.6; I.2.7 As the era of large language models (LLMs) unfolds, Preference Optimization (PO) methods have become a central approach to aligning LLMs with human preferences and improving performance. We propose Maximum a Posteriori Preference Optimization (MaPPO), a methodology for learning from preferences that explicitly incorporates prior reward knowledge into the optimization objective. Building on the paradigm employed by Direct Preference Optimization (DPO) and its variants of treating preference learning as a Maximum Likelihood Estimation (MLE) problem, MaPPO integrates prior reward estimates into a principled Maximum a Posteriori (MaP) objective. This not only generalizes DPO and its variants, but also enhances alignment by mitigating the oversimplified binary classification of responses. Additionally, MaPPO introduces no additional hyperparameters, and supports preference optimization in both offline and online settings. In addition, MaPPO can be used as a plugin for DPO variants, including widely used SimPO, IPO and CPO, and produce consistent improvements. Extensive empirical evaluations of different model sizes and model series on three standard benchmarks (MT-Bench, AlpacaEval 2.0, and Arena-Hard) demonstrate consistent improvements in alignment performance without sacrificing computational efficiency. |
| title | MaPPO: Maximum a Posteriori Preference Optimization with Prior Knowledge |
| topic | Machine Learning Artificial Intelligence Computation and Language I.2.6; I.2.7 |
| url | https://arxiv.org/abs/2507.21183 |