General Scales Unlock AI Evaluation with Explanatory and Predictive Power
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910876594864128 |
|---|---|
| author | Zhou, Lexin Pacchiardi, Lorenzo Martínez-Plumed, Fernando Collins, Katherine M. Moros-Daval, Yael Zhang, Seraphina Zhao, Qinlin Huang, Yitian Sun, Luning Prunty, Jonathan E. Li, Zongqian Sánchez-García, Pablo Chen, Kexin Jiang Casares, Pablo A. M. Zu, Jiyun Burden, John Mehrbakhsh, Behzad Stillwell, David Cebrian, Manuel Wang, Jindong Henderson, Peter Wu, Sherry Tongshuang Kyllonen, Patrick C. Cheke, Lucy Xie, Xing Hernández-Orallo, José |
| author_facet | Zhou, Lexin Pacchiardi, Lorenzo Martínez-Plumed, Fernando Collins, Katherine M. Moros-Daval, Yael Zhang, Seraphina Zhao, Qinlin Huang, Yitian Sun, Luning Prunty, Jonathan E. Li, Zongqian Sánchez-García, Pablo Chen, Kexin Jiang Casares, Pablo A. M. Zu, Jiyun Burden, John Mehrbakhsh, Behzad Stillwell, David Cebrian, Manuel Wang, Jindong Henderson, Peter Wu, Sherry Tongshuang Kyllonen, Patrick C. Cheke, Lucy Xie, Xing Hernández-Orallo, José |
| contents | Ensuring safe and effective use of AI requires understanding and anticipating its performance on novel tasks, from advanced scientific challenges to transformed workplace activities. So far, benchmarking has guided progress in AI, but it has offered limited explanatory and predictive power for general-purpose AI systems, given the low transferability across diverse tasks. In this paper, we introduce general scales for AI evaluation that can explain what common AI benchmarks really measure, extract ability profiles of AI systems, and predict their performance for new task instances, in- and out-of-distribution. Our fully-automated methodology builds on 18 newly-crafted rubrics that place instance demands on general scales that do not saturate. Illustrated for 15 large language models and 63 tasks, high explanatory power is unleashed from inspecting the demand and ability profiles, bringing insights on the sensitivity and specificity exhibited by different benchmarks, and how knowledge, metacognition and reasoning are affected by model size, chain-of-thought and distillation. Surprisingly, high predictive power at the instance level becomes possible using these demand levels, providing superior estimates over black-box baseline predictors based on embeddings or finetuning, especially in out-of-distribution settings (new tasks and new benchmarks). The scales, rubrics, battery, techniques and results presented here represent a major step for AI evaluation, underpinning the reliable deployment of AI in the years ahead. (Collaborative platform: https://kinds-of-intelligence-cfi.github.io/ADELE.) |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_06378 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | General Scales Unlock AI Evaluation with Explanatory and Predictive Power Zhou, Lexin Pacchiardi, Lorenzo Martínez-Plumed, Fernando Collins, Katherine M. Moros-Daval, Yael Zhang, Seraphina Zhao, Qinlin Huang, Yitian Sun, Luning Prunty, Jonathan E. Li, Zongqian Sánchez-García, Pablo Chen, Kexin Jiang Casares, Pablo A. M. Zu, Jiyun Burden, John Mehrbakhsh, Behzad Stillwell, David Cebrian, Manuel Wang, Jindong Henderson, Peter Wu, Sherry Tongshuang Kyllonen, Patrick C. Cheke, Lucy Xie, Xing Hernández-Orallo, José Artificial Intelligence Computation and Language Computers and Society Ensuring safe and effective use of AI requires understanding and anticipating its performance on novel tasks, from advanced scientific challenges to transformed workplace activities. So far, benchmarking has guided progress in AI, but it has offered limited explanatory and predictive power for general-purpose AI systems, given the low transferability across diverse tasks. In this paper, we introduce general scales for AI evaluation that can explain what common AI benchmarks really measure, extract ability profiles of AI systems, and predict their performance for new task instances, in- and out-of-distribution. Our fully-automated methodology builds on 18 newly-crafted rubrics that place instance demands on general scales that do not saturate. Illustrated for 15 large language models and 63 tasks, high explanatory power is unleashed from inspecting the demand and ability profiles, bringing insights on the sensitivity and specificity exhibited by different benchmarks, and how knowledge, metacognition and reasoning are affected by model size, chain-of-thought and distillation. Surprisingly, high predictive power at the instance level becomes possible using these demand levels, providing superior estimates over black-box baseline predictors based on embeddings or finetuning, especially in out-of-distribution settings (new tasks and new benchmarks). The scales, rubrics, battery, techniques and results presented here represent a major step for AI evaluation, underpinning the reliable deployment of AI in the years ahead. (Collaborative platform: https://kinds-of-intelligence-cfi.github.io/ADELE.) |
| title | General Scales Unlock AI Evaluation with Explanatory and Predictive Power |
| topic | Artificial Intelligence Computation and Language Computers and Society |
| url | https://arxiv.org/abs/2503.06378 |