ParetoHqD: Fast Offline Multiobjective Alignment of Large Language Models using Pareto High-quality Data
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915699023151104 |
|---|---|
| author | Gu, Haoran Wang, Handing Mei, Yi Zhang, Mengjie Jin, Yaochu |
| author_facet | Gu, Haoran Wang, Handing Mei, Yi Zhang, Mengjie Jin, Yaochu |
| contents | Aligning large language models with multiple human expectations and values is crucial for ensuring that they adequately serve a variety of user needs. To this end, offline multiobjective alignment algorithms such as the Rewards-in-Context algorithm have shown strong performance and efficiency. However, inappropriate preference representations and training with imbalanced reward scores limit the performance of such algorithms. In this work, we introduce ParetoHqD that addresses the above issues by representing human preferences as preference directions in the objective space and regarding data near the Pareto front as "high-quality" data. For each preference, ParetoHqD follows a two-stage supervised fine-tuning process, where each stage uses an individual Pareto high-quality training set that best matches its preference direction. The experimental results have demonstrated the superiority of ParetoHqD over five baselines on two multiobjective alignment tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_16628 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ParetoHqD: Fast Offline Multiobjective Alignment of Large Language Models using Pareto High-quality Data Gu, Haoran Wang, Handing Mei, Yi Zhang, Mengjie Jin, Yaochu Machine Learning Computation and Language Aligning large language models with multiple human expectations and values is crucial for ensuring that they adequately serve a variety of user needs. To this end, offline multiobjective alignment algorithms such as the Rewards-in-Context algorithm have shown strong performance and efficiency. However, inappropriate preference representations and training with imbalanced reward scores limit the performance of such algorithms. In this work, we introduce ParetoHqD that addresses the above issues by representing human preferences as preference directions in the objective space and regarding data near the Pareto front as "high-quality" data. For each preference, ParetoHqD follows a two-stage supervised fine-tuning process, where each stage uses an individual Pareto high-quality training set that best matches its preference direction. The experimental results have demonstrated the superiority of ParetoHqD over five baselines on two multiobjective alignment tasks. |
| title | ParetoHqD: Fast Offline Multiobjective Alignment of Large Language Models using Pareto High-quality Data |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2504.16628 |