Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2310.16379 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909056580452352 |
|---|---|
| author | Wang, Xiting Jiang, Liming Hernandez-Orallo, Jose Stillwell, David Sun, Luning Luo, Fang Xie, Xing |
| author_facet | Wang, Xiting Jiang, Liming Hernandez-Orallo, Jose Stillwell, David Sun, Luning Luo, Fang Xie, Xing |
| contents | Comprehensive and accurate evaluation of general-purpose AI systems such as large language models allows for effective mitigation of their risks and deepened understanding of their capabilities. Current evaluation methodology, mostly based on benchmarks of specific tasks, falls short of adequately assessing these versatile AI systems, as present techniques lack a scientific foundation for predicting their performance on unforeseen tasks and explaining their varying performance on specific task items or user inputs. Moreover, existing benchmarks of specific tasks raise growing concerns about their reliability and validity. To tackle these challenges, we suggest transitioning from task-oriented evaluation to construct-oriented evaluation. Psychometrics, the science of psychological measurement, provides a rigorous methodology for identifying and measuring the latent constructs that underlie performance across multiple tasks. We discuss its merits, warn against potential pitfalls, and propose a framework to put it into practice. Finally, we explore future opportunities of integrating psychometrics with the evaluation of general-purpose AI systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_16379 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Evaluating General-Purpose AI with Psychometrics Wang, Xiting Jiang, Liming Hernandez-Orallo, Jose Stillwell, David Sun, Luning Luo, Fang Xie, Xing Artificial Intelligence Computers and Society Comprehensive and accurate evaluation of general-purpose AI systems such as large language models allows for effective mitigation of their risks and deepened understanding of their capabilities. Current evaluation methodology, mostly based on benchmarks of specific tasks, falls short of adequately assessing these versatile AI systems, as present techniques lack a scientific foundation for predicting their performance on unforeseen tasks and explaining their varying performance on specific task items or user inputs. Moreover, existing benchmarks of specific tasks raise growing concerns about their reliability and validity. To tackle these challenges, we suggest transitioning from task-oriented evaluation to construct-oriented evaluation. Psychometrics, the science of psychological measurement, provides a rigorous methodology for identifying and measuring the latent constructs that underlie performance across multiple tasks. We discuss its merits, warn against potential pitfalls, and propose a framework to put it into practice. Finally, we explore future opportunities of integrating psychometrics with the evaluation of general-purpose AI systems. |
| title | Evaluating General-Purpose AI with Psychometrics |
| topic | Artificial Intelligence Computers and Society |
| url | https://arxiv.org/abs/2310.16379 |