From traces to measures: Large language models as a tool for psychological measurement from text
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916436016889856 |
|---|---|
| author | Simons, Joseph J. P. Ze, Wong Liang Bhattacharya, Prasanta Loh, Brandon Siyuan Gao, Wei |
| author_facet | Simons, Joseph J. P. Ze, Wong Liang Bhattacharya, Prasanta Loh, Brandon Siyuan Gao, Wei |
| contents | Large language models are increasingly being used to label or rate psychological features in text data. This approach helps address one of the limiting factors of digital trace data - their lack of an inherent target of measurement. However, this approach is also a form of psychological measurement (using observable variables to quantify a hypothetical latent construct). As such, these ratings are subject to the same psychometric considerations of reliability and validity as more standard psychological measures. Here we present a workflow for developing and evaluating large language model based measures of psychological features which incorporate these considerations. We also provide an example, attempting to measure the previously established constructs of attitude certainty, importance and moralization from text. Using a pool of prompts adapted from existing measurement instruments, we find they have good levels of internal consistency but only partially meet validity criteria. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_07447 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | From traces to measures: Large language models as a tool for psychological measurement from text Simons, Joseph J. P. Ze, Wong Liang Bhattacharya, Prasanta Loh, Brandon Siyuan Gao, Wei Human-Computer Interaction Large language models are increasingly being used to label or rate psychological features in text data. This approach helps address one of the limiting factors of digital trace data - their lack of an inherent target of measurement. However, this approach is also a form of psychological measurement (using observable variables to quantify a hypothetical latent construct). As such, these ratings are subject to the same psychometric considerations of reliability and validity as more standard psychological measures. Here we present a workflow for developing and evaluating large language model based measures of psychological features which incorporate these considerations. We also provide an example, attempting to measure the previously established constructs of attitude certainty, importance and moralization from text. Using a pool of prompts adapted from existing measurement instruments, we find they have good levels of internal consistency but only partially meet validity criteria. |
| title | From traces to measures: Large language models as a tool for psychological measurement from text |
| topic | Human-Computer Interaction |
| url | https://arxiv.org/abs/2405.07447 |