AI Psychometrics: Evaluating the Psychological Reasoning of Large Language Models with Psychometric Validities
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866911511232905216 |
|---|---|
| author | Li, Yibai Lin, Xiaolin Sha, Zhenghui Jin, Zhiye Li, Xiaobing |
| author_facet | Li, Yibai Lin, Xiaolin Sha, Zhenghui Jin, Zhiye Li, Xiaobing |
| contents | The immense number of parameters and deep neural networks make large language models (LLMs) rival the complexity of human brains, which also makes them opaque ``black box'' systems that are challenging to evaluate and interpret. AI Psychometrics is an emerging field that aims to tackle these challenges by applying psychometric methodologies to evaluate and interpret the psychological traits and processes of artificial intelligence (AI) systems. This paper investigates the application of AI Psychometrics to evaluate the psychological reasoning and overall psychometric validity of four prominent LLMs: GPT-3.5, GPT-4, LLaMA-2, and LLaMA-3. Using the Technology Acceptance Model (TAM), we examined convergent, discriminant, predictive, and external validity across these models. Our findings reveal that the responses from all these models generally met all validity criteria. Moreover, higher-performing models like GPT-4 and LLaMA-3 consistently demonstrated superior psychometric validity compared to their predecessors, GPT-3.5 and LLaMA-2. These results help to establish the validity of applying AI Psychometrics to evaluate and interpret large language models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_11279 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | AI Psychometrics: Evaluating the Psychological Reasoning of Large Language Models with Psychometric Validities Li, Yibai Lin, Xiaolin Sha, Zhenghui Jin, Zhiye Li, Xiaobing Artificial Intelligence 68T50, 62P15, 91E45 I.2.7; I.2.0; J.4; H.1.2 The immense number of parameters and deep neural networks make large language models (LLMs) rival the complexity of human brains, which also makes them opaque ``black box'' systems that are challenging to evaluate and interpret. AI Psychometrics is an emerging field that aims to tackle these challenges by applying psychometric methodologies to evaluate and interpret the psychological traits and processes of artificial intelligence (AI) systems. This paper investigates the application of AI Psychometrics to evaluate the psychological reasoning and overall psychometric validity of four prominent LLMs: GPT-3.5, GPT-4, LLaMA-2, and LLaMA-3. Using the Technology Acceptance Model (TAM), we examined convergent, discriminant, predictive, and external validity across these models. Our findings reveal that the responses from all these models generally met all validity criteria. Moreover, higher-performing models like GPT-4 and LLaMA-3 consistently demonstrated superior psychometric validity compared to their predecessors, GPT-3.5 and LLaMA-2. These results help to establish the validity of applying AI Psychometrics to evaluate and interpret large language models. |
| title | AI Psychometrics: Evaluating the Psychological Reasoning of Large Language Models with Psychometric Validities |
| topic | Artificial Intelligence 68T50, 62P15, 91E45 I.2.7; I.2.0; J.4; H.1.2 |
| url | https://arxiv.org/abs/2603.11279 |