A Detailed Factor Analysis for the Political Compass Test: Navigating Ideologies of Large Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914151521058816 |
|---|---|
| author | Kamal, Sadia Prakash, Lalu Prasad Yadav Rafiuddin, S M Rakib, Mohammed Sen, Atriya Choudhury, Sagnik Ray |
| author_facet | Kamal, Sadia Prakash, Lalu Prasad Yadav Rafiuddin, S M Rakib, Mohammed Sen, Atriya Choudhury, Sagnik Ray |
| contents | The Political Compass Test (PCT) and similar surveys are commonly used to assess political bias in auto-regressive LLMs. Our rigorous statistical experiments show that while changes to standard generation parameters have minimal effect on PCT scores, prompt phrasing and fine-tuning individually and together can significantly influence results. Interestingly, fine-tuning on politically rich vs. neutral datasets does not lead to different shifts in scores. We also generalize these findings to a similar popular test called 8 Values. Humans do not change their responses to questions when prompted differently (``answer this question'' vs ``state your opinion''), or after exposure to politically neutral text, such as mathematical formulae. But the fact that the models do so raises concerns about the validity of these tests for measuring model bias, and paves the way for deeper exploration into how political and social views are encoded in LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_22493 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | A Detailed Factor Analysis for the Political Compass Test: Navigating Ideologies of Large Language Models Kamal, Sadia Prakash, Lalu Prasad Yadav Rafiuddin, S M Rakib, Mohammed Sen, Atriya Choudhury, Sagnik Ray Computers and Society Computation and Language Machine Learning The Political Compass Test (PCT) and similar surveys are commonly used to assess political bias in auto-regressive LLMs. Our rigorous statistical experiments show that while changes to standard generation parameters have minimal effect on PCT scores, prompt phrasing and fine-tuning individually and together can significantly influence results. Interestingly, fine-tuning on politically rich vs. neutral datasets does not lead to different shifts in scores. We also generalize these findings to a similar popular test called 8 Values. Humans do not change their responses to questions when prompted differently (``answer this question'' vs ``state your opinion''), or after exposure to politically neutral text, such as mathematical formulae. But the fact that the models do so raises concerns about the validity of these tests for measuring model bias, and paves the way for deeper exploration into how political and social views are encoded in LLMs. |
| title | A Detailed Factor Analysis for the Political Compass Test: Navigating Ideologies of Large Language Models |
| topic | Computers and Society Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2506.22493 |