UC-MOA: Utility-Conditioned Multi-Objective Alignment for Distributional Pareto-Optimality
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916741441912832 |
|---|---|
| author | Cheng, Zelei Cai, Xin-Qiang Tang, Yuting Zhang, Pushi Yang, Boming Sugiyama, Masashi Xing, Xinyu |
| author_facet | Cheng, Zelei Cai, Xin-Qiang Tang, Yuting Zhang, Pushi Yang, Boming Sugiyama, Masashi Xing, Xinyu |
| contents | Reinforcement Learning from Human Feedback (RLHF) has become a cornerstone for aligning large language models (LLMs) with human values. However, existing approaches struggle to capture the multi-dimensional, distributional nuances of human preferences. Methods such as RiC that directly inject raw reward values into prompts face significant numerical sensitivity issues--for instance, LLMs may fail to distinguish between 9.11 and 9.8--while alternatives like MORLHF, Rewarded Soups, and MODPO incur high computational costs by training multiple models. In this work, we introduce Utility-Conditioned Multi-Objective Alignment (UC-MOA), a novel framework that overcomes these limitations. Our approach leverages a diverse set of strictly increasing, non-linear utility functions to transform user-specified preferences into symbolic tokens, which are then used to condition a single LLM. This design not only mitigates numerical reasoning challenges but also substantially reduces training overhead, yielding models that achieve superior Pareto fronts and robust alignment across complex reward dimensions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_10669 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | UC-MOA: Utility-Conditioned Multi-Objective Alignment for Distributional Pareto-Optimality Cheng, Zelei Cai, Xin-Qiang Tang, Yuting Zhang, Pushi Yang, Boming Sugiyama, Masashi Xing, Xinyu Computation and Language Artificial Intelligence Reinforcement Learning from Human Feedback (RLHF) has become a cornerstone for aligning large language models (LLMs) with human values. However, existing approaches struggle to capture the multi-dimensional, distributional nuances of human preferences. Methods such as RiC that directly inject raw reward values into prompts face significant numerical sensitivity issues--for instance, LLMs may fail to distinguish between 9.11 and 9.8--while alternatives like MORLHF, Rewarded Soups, and MODPO incur high computational costs by training multiple models. In this work, we introduce Utility-Conditioned Multi-Objective Alignment (UC-MOA), a novel framework that overcomes these limitations. Our approach leverages a diverse set of strictly increasing, non-linear utility functions to transform user-specified preferences into symbolic tokens, which are then used to condition a single LLM. This design not only mitigates numerical reasoning challenges but also substantially reduces training overhead, yielding models that achieve superior Pareto fronts and robust alignment across complex reward dimensions. |
| title | UC-MOA: Utility-Conditioned Multi-Objective Alignment for Distributional Pareto-Optimality |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2503.10669 |