Identifying good forecasters via adaptive cognitive tests
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911040153845760 |
|---|---|
| author | Merkle, Edgar C. Petrov, Nikolay Zhu, Sophie Ma Karger, Ezra Tetlock, Philip E. Himmelstein, Mark |
| author_facet | Merkle, Edgar C. Petrov, Nikolay Zhu, Sophie Ma Karger, Ezra Tetlock, Philip E. Himmelstein, Mark |
| contents | Assessing forecasting performance is a time intensive activity, often requiring months or years before we know whether or not the reported forecasts were accurate. Cognitive tests can be quickly administered and are predictive of forecasting performance, but it is unclear which and how many tests are optimal. In this study, we develop adaptive cognitive tests that optimize the selection and efficiency of cognitive tests to assess forecasters of different skill levels. The tests are based on item response models and the adaptive testing procedures commonly used in educational testing. We show how the procedures can select highly informative cognitive tests from a larger battery of tests, thereby reducing the time taken to administer the tests. We use a second, independent dataset to show that the selected tests yield scores that are highly related to out-of-sample forecasting performance. The approach enables real-time, adaptive testing, providing immediate insights into forecasting talent in practical contexts. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_11126 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Identifying good forecasters via adaptive cognitive tests Merkle, Edgar C. Petrov, Nikolay Zhu, Sophie Ma Karger, Ezra Tetlock, Philip E. Himmelstein, Mark Applications Assessing forecasting performance is a time intensive activity, often requiring months or years before we know whether or not the reported forecasts were accurate. Cognitive tests can be quickly administered and are predictive of forecasting performance, but it is unclear which and how many tests are optimal. In this study, we develop adaptive cognitive tests that optimize the selection and efficiency of cognitive tests to assess forecasters of different skill levels. The tests are based on item response models and the adaptive testing procedures commonly used in educational testing. We show how the procedures can select highly informative cognitive tests from a larger battery of tests, thereby reducing the time taken to administer the tests. We use a second, independent dataset to show that the selected tests yield scores that are highly related to out-of-sample forecasting performance. The approach enables real-time, adaptive testing, providing immediate insights into forecasting talent in practical contexts. |
| title | Identifying good forecasters via adaptive cognitive tests |
| topic | Applications |
| url | https://arxiv.org/abs/2411.11126 |