Direct Alignment with Heterogeneous Preferences
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913703293616128 |
|---|---|
| author | Shirali, Ali Nasr-Esfahany, Arash Alomar, Abdullah Mirtaheri, Parsa Abebe, Rediet Procaccia, Ariel |
| author_facet | Shirali, Ali Nasr-Esfahany, Arash Alomar, Abdullah Mirtaheri, Parsa Abebe, Rediet Procaccia, Ariel |
| contents | Alignment with human preferences is commonly framed using a universal reward function, even though human preferences are inherently heterogeneous. We formalize this heterogeneity by introducing user types and examine the limits of the homogeneity assumption. We show that aligning to heterogeneous preferences with a single policy is best achieved using the average reward across user types. However, this requires additional information about annotators. We examine improvements under different information settings, focusing on direct alignment methods. We find that minimal information can yield first-order improvements, while full feedback from each user type leads to consistent learning of the optimal policy. Surprisingly, however, no sample-efficient consistent direct loss exists in this latter setting. These results reveal a fundamental tension between consistency and sample efficiency in direct policy alignment. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_16320 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Direct Alignment with Heterogeneous Preferences Shirali, Ali Nasr-Esfahany, Arash Alomar, Abdullah Mirtaheri, Parsa Abebe, Rediet Procaccia, Ariel Artificial Intelligence Machine Learning Alignment with human preferences is commonly framed using a universal reward function, even though human preferences are inherently heterogeneous. We formalize this heterogeneity by introducing user types and examine the limits of the homogeneity assumption. We show that aligning to heterogeneous preferences with a single policy is best achieved using the average reward across user types. However, this requires additional information about annotators. We examine improvements under different information settings, focusing on direct alignment methods. We find that minimal information can yield first-order improvements, while full feedback from each user type leads to consistent learning of the optimal policy. Surprisingly, however, no sample-efficient consistent direct loss exists in this latter setting. These results reveal a fundamental tension between consistency and sample efficiency in direct policy alignment. |
| title | Direct Alignment with Heterogeneous Preferences |
| topic | Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2502.16320 |