PerPO: Perceptual Preference Optimization via Discriminative Rewarding
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866929701075812352 |
|---|---|
| author | Zhu, Zining Zhao, Liang Lin, Kangheng Yang, Jinze Yu, En Liu, Chenglong Wei, Haoran Sun, Jianjian Ge, Zheng Zhang, Xiangyu |
| author_facet | Zhu, Zining Zhao, Liang Lin, Kangheng Yang, Jinze Yu, En Liu, Chenglong Wei, Haoran Sun, Jianjian Ge, Zheng Zhang, Xiangyu |
| contents | This paper presents Perceptual Preference Optimization (PerPO), a perception alignment method aimed at addressing the visual discrimination challenges in generative pre-trained multimodal large language models (MLLMs). To align MLLMs with human visual perception process, PerPO employs discriminative rewarding to gather diverse negative samples, followed by listwise preference optimization to rank them.By utilizing the reward as a quantitative margin for ranking, our method effectively bridges generative preference optimization and discriminative empirical risk minimization. PerPO significantly enhances MLLMs' visual discrimination capabilities while maintaining their generative strengths, mitigates image-unconditional reward hacking, and ensures consistent performance across visual tasks. This work marks a crucial step towards more perceptually aligned and versatile MLLMs. We also hope that PerPO will encourage the community to rethink MLLM alignment strategies. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_04371 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | PerPO: Perceptual Preference Optimization via Discriminative Rewarding Zhu, Zining Zhao, Liang Lin, Kangheng Yang, Jinze Yu, En Liu, Chenglong Wei, Haoran Sun, Jianjian Ge, Zheng Zhang, Xiangyu Artificial Intelligence Computation and Language Machine Learning This paper presents Perceptual Preference Optimization (PerPO), a perception alignment method aimed at addressing the visual discrimination challenges in generative pre-trained multimodal large language models (MLLMs). To align MLLMs with human visual perception process, PerPO employs discriminative rewarding to gather diverse negative samples, followed by listwise preference optimization to rank them.By utilizing the reward as a quantitative margin for ranking, our method effectively bridges generative preference optimization and discriminative empirical risk minimization. PerPO significantly enhances MLLMs' visual discrimination capabilities while maintaining their generative strengths, mitigates image-unconditional reward hacking, and ensures consistent performance across visual tasks. This work marks a crucial step towards more perceptually aligned and versatile MLLMs. We also hope that PerPO will encourage the community to rethink MLLM alignment strategies. |
| title | PerPO: Perceptual Preference Optimization via Discriminative Rewarding |
| topic | Artificial Intelligence Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2502.04371 |