PLOT: Enhancing Preference Learning via Optimal Transport
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917380159963136 |
|---|---|
| author | Zhu, Liang Bai, Yuelin Ren, Xiankun Yang, Jiaxi Zhang, Lei Fang, Feiteng Alinejad-Rokny, Hamid Tan, Minghuan Yang, Min |
| author_facet | Zhu, Liang Bai, Yuelin Ren, Xiankun Yang, Jiaxi Zhang, Lei Fang, Feiteng Alinejad-Rokny, Hamid Tan, Minghuan Yang, Min |
| contents | Preference learning in Large Language Models (LLMs) has advanced significantly, yet existing methods remain limited by modest performance gains, high computational costs, hyperparameter sensitivity, and insufficient modeling of global token-level relationships. We introduce PLOT, which enhances Preference Learning in fine-tuning-based alignment through a token-level loss derived from Optimal Transport. By formulating preference learning as an Optimal Transport Problem, PLOT aligns model outputs with human preferences while preserving the original distribution of LLMs, ensuring stability and robustness. Furthermore, PLOT leverages token embeddings to capture semantic relationships, enabling globally informed optimization. Experiments across two preference categories - Human Values and Logic & Problem Solving - spanning seven subpreferences demonstrate that PLOT consistently improves alignment performance while maintaining fluency and coherence. These results substantiate optimal transport as a principled methodology for preference learning, establishing a theoretically grounded framework that provides new insights for preference learning of LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_01837 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | PLOT: Enhancing Preference Learning via Optimal Transport Zhu, Liang Bai, Yuelin Ren, Xiankun Yang, Jiaxi Zhang, Lei Fang, Feiteng Alinejad-Rokny, Hamid Tan, Minghuan Yang, Min Computation and Language Preference learning in Large Language Models (LLMs) has advanced significantly, yet existing methods remain limited by modest performance gains, high computational costs, hyperparameter sensitivity, and insufficient modeling of global token-level relationships. We introduce PLOT, which enhances Preference Learning in fine-tuning-based alignment through a token-level loss derived from Optimal Transport. By formulating preference learning as an Optimal Transport Problem, PLOT aligns model outputs with human preferences while preserving the original distribution of LLMs, ensuring stability and robustness. Furthermore, PLOT leverages token embeddings to capture semantic relationships, enabling globally informed optimization. Experiments across two preference categories - Human Values and Logic & Problem Solving - spanning seven subpreferences demonstrate that PLOT consistently improves alignment performance while maintaining fluency and coherence. These results substantiate optimal transport as a principled methodology for preference learning, establishing a theoretically grounded framework that provides new insights for preference learning of LLMs. |
| title | PLOT: Enhancing Preference Learning via Optimal Transport |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2604.01837 |