Are generative AI text annotations systematically biased?
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917134343340032 |
|---|---|
| author | Stolwijk, Sjoerd B. Boukes, Mark Trilling, Damian |
| author_facet | Stolwijk, Sjoerd B. Boukes, Mark Trilling, Damian |
| contents | This paper investigates bias in GLLM annotations by conceptually replicating manual annotations of Boukes (2024). Using various GLLMs (Llama3.1:8b, Llama3.3:70b, GPT4o, Qwen2.5:72b) in combination with five different prompts for five concepts (political content, interactivity, rationality, incivility, and ideology). We find GLLMs perform adequate in terms of F1 scores, but differ from manual annotations in terms of prevalence, yield substantively different downstream results, and display systematic bias in that they overlap more with each other than with manual annotations. Differences in F1 scores fail to account for the degree of bias. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_08404 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Are generative AI text annotations systematically biased? Stolwijk, Sjoerd B. Boukes, Mark Trilling, Damian Computation and Language Artificial Intelligence This paper investigates bias in GLLM annotations by conceptually replicating manual annotations of Boukes (2024). Using various GLLMs (Llama3.1:8b, Llama3.3:70b, GPT4o, Qwen2.5:72b) in combination with five different prompts for five concepts (political content, interactivity, rationality, incivility, and ideology). We find GLLMs perform adequate in terms of F1 scores, but differ from manual annotations in terms of prevalence, yield substantively different downstream results, and display systematic bias in that they overlap more with each other than with manual annotations. Differences in F1 scores fail to account for the degree of bias. |
| title | Are generative AI text annotations systematically biased? |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2512.08404 |