Measure what Matters: Psychometric Evaluation of AI with Situational Judgment Tests
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913105414455296 |
|---|---|
| author | Yost, Alexandra Jain, Shreyans Raval, Shivam Corser, Grant Roush, Allen Xu, Nina Hammack, Jacqueline Shwartz-Ziv, Ravid Abdullah, Amirali |
| author_facet | Yost, Alexandra Jain, Shreyans Raval, Shivam Corser, Grant Roush, Allen Xu, Nina Hammack, Jacqueline Shwartz-Ziv, Ravid Abdullah, Amirali |
| contents | Persona conditioning is widely used to steer large language model (LLM) behavior, but it is unclear whether it induces stable behavioral structure or superficial variation. We propose a framework to measure consistent behavioral tendencies using situational judgment tests (SJTs), multidimensional item response theory (MIRT), and structured synthetic personas, treating responses as observations of latent behavioral variables.
Across large-scale SJT and persona datasets, we find that persona-conditioned behaviors are stable across runs, latent trait scores predict external benchmarks (e.g., TruthfulQA, EmoBench), and MIRT reveals consistent latent structure. We validate these results through human annotation, benchmark evaluation, and internal consistency analyses.
We interpret these traits not as human personality, but as stable behavioral tendencies expressed across contexts. Our results show that scenario-based psychometric evaluation provides a more reliable alternative to classical self-report approaches for assessing LLM behavior, and we release datasets to support further study. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_22170 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Measure what Matters: Psychometric Evaluation of AI with Situational Judgment Tests Yost, Alexandra Jain, Shreyans Raval, Shivam Corser, Grant Roush, Allen Xu, Nina Hammack, Jacqueline Shwartz-Ziv, Ravid Abdullah, Amirali Artificial Intelligence I.2.7; I.2.6; H.1.2; J.4 Persona conditioning is widely used to steer large language model (LLM) behavior, but it is unclear whether it induces stable behavioral structure or superficial variation. We propose a framework to measure consistent behavioral tendencies using situational judgment tests (SJTs), multidimensional item response theory (MIRT), and structured synthetic personas, treating responses as observations of latent behavioral variables. Across large-scale SJT and persona datasets, we find that persona-conditioned behaviors are stable across runs, latent trait scores predict external benchmarks (e.g., TruthfulQA, EmoBench), and MIRT reveals consistent latent structure. We validate these results through human annotation, benchmark evaluation, and internal consistency analyses. We interpret these traits not as human personality, but as stable behavioral tendencies expressed across contexts. Our results show that scenario-based psychometric evaluation provides a more reliable alternative to classical self-report approaches for assessing LLM behavior, and we release datasets to support further study. |
| title | Measure what Matters: Psychometric Evaluation of AI with Situational Judgment Tests |
| topic | Artificial Intelligence I.2.7; I.2.6; H.1.2; J.4 |
| url | https://arxiv.org/abs/2510.22170 |