Steerable Pluralism: Pluralistic Alignment via Few-Shot Comparative Regression
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912532922368000 |
|---|---|
| author | Adams, Jadie Hu, Brian Veenhuis, Emily Joy, David Ravichandran, Bharadwaj Bray, Aaron Hoogs, Anthony Basharat, Arslan |
| author_facet | Adams, Jadie Hu, Brian Veenhuis, Emily Joy, David Ravichandran, Bharadwaj Bray, Aaron Hoogs, Anthony Basharat, Arslan |
| contents | Large language models (LLMs) are currently aligned using techniques such as reinforcement learning from human feedback (RLHF). However, these methods use scalar rewards that can only reflect user preferences on average. Pluralistic alignment instead seeks to capture diverse user preferences across a set of attributes, moving beyond just helpfulness and harmlessness. Toward this end, we propose a steerable pluralistic model based on few-shot comparative regression that can adapt to individual user preferences. Our approach leverages in-context learning and reasoning, grounded in a set of fine-grained attributes, to compare response options and make aligned choices. To evaluate our algorithm, we also propose two new steerable pluralistic benchmarks by adapting the Moral Integrity Corpus (MIC) and the HelpSteer2 datasets, demonstrating the applicability of our approach to value-aligned decision-making and reward modeling, respectively. Our few-shot comparative regression approach is interpretable and compatible with different attributes and LLMs, while outperforming multiple baseline and state-of-the-art methods. Our work provides new insights and research directions in pluralistic alignment, enabling a more fair and representative use of LLMs and advancing the state-of-the-art in ethical AI. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_08509 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Steerable Pluralism: Pluralistic Alignment via Few-Shot Comparative Regression Adams, Jadie Hu, Brian Veenhuis, Emily Joy, David Ravichandran, Bharadwaj Bray, Aaron Hoogs, Anthony Basharat, Arslan Computation and Language Artificial Intelligence Large language models (LLMs) are currently aligned using techniques such as reinforcement learning from human feedback (RLHF). However, these methods use scalar rewards that can only reflect user preferences on average. Pluralistic alignment instead seeks to capture diverse user preferences across a set of attributes, moving beyond just helpfulness and harmlessness. Toward this end, we propose a steerable pluralistic model based on few-shot comparative regression that can adapt to individual user preferences. Our approach leverages in-context learning and reasoning, grounded in a set of fine-grained attributes, to compare response options and make aligned choices. To evaluate our algorithm, we also propose two new steerable pluralistic benchmarks by adapting the Moral Integrity Corpus (MIC) and the HelpSteer2 datasets, demonstrating the applicability of our approach to value-aligned decision-making and reward modeling, respectively. Our few-shot comparative regression approach is interpretable and compatible with different attributes and LLMs, while outperforming multiple baseline and state-of-the-art methods. Our work provides new insights and research directions in pluralistic alignment, enabling a more fair and representative use of LLMs and advancing the state-of-the-art in ethical AI. |
| title | Steerable Pluralism: Pluralistic Alignment via Few-Shot Comparative Regression |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2508.08509 |