Human- vs. AI-generated tests: dimensionality and information accuracy in latent trait evaluation
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866908830877614080 |
|---|---|
| author | Angelelli, Mario Oliva, Morena Arima, Serena Ciavolino, Enrico |
| author_facet | Angelelli, Mario Oliva, Morena Arima, Serena Ciavolino, Enrico |
| contents | Artificial Intelligence (AI) and large language models (LLMs) are increasingly used in social and psychological research. Among potential applications, LLMs can be used to generate, customise, or adapt measurement instruments. This study presents a preliminary investigation of AI-generated questionnaires by comparing two ChatGPT-based adaptations of the Body Awareness Questionnaire (BAQ) with the validated human-developed version. The AI instruments were designed with different levels of explicitness in content and instructions on construct facets, and their psychometric properties were assessed using a Bayesian Graded Response Model. Results show that although surface wording between AI and original items was similar, differences emerged in dimensionality and in the distribution of item and test information across latent traits. These findings illustrate the importance of applying statistical measures of accuracy to ensure the validity and interpretability of AI-driven tools. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_24739 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Human- vs. AI-generated tests: dimensionality and information accuracy in latent trait evaluation Angelelli, Mario Oliva, Morena Arima, Serena Ciavolino, Enrico Human-Computer Interaction Information Theory Methodology 62Bxx, 62F15, 62P15 Artificial Intelligence (AI) and large language models (LLMs) are increasingly used in social and psychological research. Among potential applications, LLMs can be used to generate, customise, or adapt measurement instruments. This study presents a preliminary investigation of AI-generated questionnaires by comparing two ChatGPT-based adaptations of the Body Awareness Questionnaire (BAQ) with the validated human-developed version. The AI instruments were designed with different levels of explicitness in content and instructions on construct facets, and their psychometric properties were assessed using a Bayesian Graded Response Model. Results show that although surface wording between AI and original items was similar, differences emerged in dimensionality and in the distribution of item and test information across latent traits. These findings illustrate the importance of applying statistical measures of accuracy to ensure the validity and interpretability of AI-driven tools. |
| title | Human- vs. AI-generated tests: dimensionality and information accuracy in latent trait evaluation |
| topic | Human-Computer Interaction Information Theory Methodology 62Bxx, 62F15, 62P15 |
| url | https://arxiv.org/abs/2510.24739 |