On the Relationship between Truth and Political Bias in Language Models
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866915039859965952 |
|---|---|
| author | Fulay, Suyash Brannon, William Mohanty, Shrestha Overney, Cassandra Poole-Dayan, Elinor Roy, Deb Kabbara, Jad |
| author_facet | Fulay, Suyash Brannon, William Mohanty, Shrestha Overney, Cassandra Poole-Dayan, Elinor Roy, Deb Kabbara, Jad |
| contents | Language model alignment research often attempts to ensure that models are not only helpful and harmless, but also truthful and unbiased. However, optimizing these objectives simultaneously can obscure how improving one aspect might impact the others. In this work, we focus on analyzing the relationship between two concepts essential in both language model alignment and political science: truthfulness and political bias. We train reward models on various popular truthfulness datasets and subsequently evaluate their political bias. Our findings reveal that optimizing reward models for truthfulness on these datasets tends to result in a left-leaning political bias. We also find that existing open-source reward models (i.e., those trained on standard human preference datasets) already show a similar bias and that the bias is larger for larger models. These results raise important questions about the datasets used to represent truthfulness, potential limitations of aligning models to be both truthful and politically unbiased, and what language models capture about the relationship between truth and politics. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_05283 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | On the Relationship between Truth and Political Bias in Language Models Fulay, Suyash Brannon, William Mohanty, Shrestha Overney, Cassandra Poole-Dayan, Elinor Roy, Deb Kabbara, Jad Computation and Language Artificial Intelligence Language model alignment research often attempts to ensure that models are not only helpful and harmless, but also truthful and unbiased. However, optimizing these objectives simultaneously can obscure how improving one aspect might impact the others. In this work, we focus on analyzing the relationship between two concepts essential in both language model alignment and political science: truthfulness and political bias. We train reward models on various popular truthfulness datasets and subsequently evaluate their political bias. Our findings reveal that optimizing reward models for truthfulness on these datasets tends to result in a left-leaning political bias. We also find that existing open-source reward models (i.e., those trained on standard human preference datasets) already show a similar bias and that the bias is larger for larger models. These results raise important questions about the datasets used to represent truthfulness, potential limitations of aligning models to be both truthful and politically unbiased, and what language models capture about the relationship between truth and politics. |
| title | On the Relationship between Truth and Political Bias in Language Models |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2409.05283 |