Beyond Demographics: Fine-tuning Large Language Models to Predict Individuals' Subjective Text Perceptions
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912251960623104 |
|---|---|
| author | Orlikowski, Matthias Pei, Jiaxin Röttger, Paul Cimiano, Philipp Jurgens, David Hovy, Dirk |
| author_facet | Orlikowski, Matthias Pei, Jiaxin Röttger, Paul Cimiano, Philipp Jurgens, David Hovy, Dirk |
| contents | People naturally vary in their annotations for subjective questions and some of this variation is thought to be due to the person's sociodemographic characteristics. LLMs have also been used to label data, but recent work has shown that models perform poorly when prompted with sociodemographic attributes, suggesting limited inherent sociodemographic knowledge. Here, we ask whether LLMs can be trained to be accurate sociodemographic models of annotator variation. Using a curated dataset of five tasks with standardized sociodemographics, we show that models do improve in sociodemographic prompting when trained but that this performance gain is largely due to models learning annotator-specific behaviour rather than sociodemographic patterns. Across all tasks, our results suggest that models learn little meaningful connection between sociodemographics and annotation, raising doubts about the current use of LLMs for simulating sociodemographic variation and behaviour. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_20897 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Beyond Demographics: Fine-tuning Large Language Models to Predict Individuals' Subjective Text Perceptions Orlikowski, Matthias Pei, Jiaxin Röttger, Paul Cimiano, Philipp Jurgens, David Hovy, Dirk Computation and Language I.2.7 People naturally vary in their annotations for subjective questions and some of this variation is thought to be due to the person's sociodemographic characteristics. LLMs have also been used to label data, but recent work has shown that models perform poorly when prompted with sociodemographic attributes, suggesting limited inherent sociodemographic knowledge. Here, we ask whether LLMs can be trained to be accurate sociodemographic models of annotator variation. Using a curated dataset of five tasks with standardized sociodemographics, we show that models do improve in sociodemographic prompting when trained but that this performance gain is largely due to models learning annotator-specific behaviour rather than sociodemographic patterns. Across all tasks, our results suggest that models learn little meaningful connection between sociodemographics and annotation, raising doubts about the current use of LLMs for simulating sociodemographic variation and behaviour. |
| title | Beyond Demographics: Fine-tuning Large Language Models to Predict Individuals' Subjective Text Perceptions |
| topic | Computation and Language I.2.7 |
| url | https://arxiv.org/abs/2502.20897 |