Inferring Capabilities from Task Performance with Bayesian Triangulation
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2023
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866915537157619712 |
|---|---|
| author | Burden, John Voudouris, Konstantinos Burnell, Ryan Rutar, Danaja Cheke, Lucy Hernández-Orallo, José |
| author_facet | Burden, John Voudouris, Konstantinos Burnell, Ryan Rutar, Danaja Cheke, Lucy Hernández-Orallo, José |
| contents | As machine learning models become more general, we need to characterise them in richer, more meaningful ways. We describe a method to infer the cognitive profile of a system from diverse experimental data. To do so, we introduce measurement layouts that model how task-instance features interact with system capabilities to affect performance. These features must be triangulated in complex ways to be able to infer capabilities from non-populational data -- a challenge for traditional psychometric and inferential tools. Using the Bayesian probabilistic programming library PyMC, we infer different cognitive profiles for agents in two scenarios: 68 actual contestants in the AnimalAI Olympics and 30 synthetic agents for O-PIAAGETS, an object permanence battery. We showcase the potential for capability-oriented evaluation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2309_11975 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Inferring Capabilities from Task Performance with Bayesian Triangulation Burden, John Voudouris, Konstantinos Burnell, Ryan Rutar, Danaja Cheke, Lucy Hernández-Orallo, José Artificial Intelligence As machine learning models become more general, we need to characterise them in richer, more meaningful ways. We describe a method to infer the cognitive profile of a system from diverse experimental data. To do so, we introduce measurement layouts that model how task-instance features interact with system capabilities to affect performance. These features must be triangulated in complex ways to be able to infer capabilities from non-populational data -- a challenge for traditional psychometric and inferential tools. Using the Bayesian probabilistic programming library PyMC, we infer different cognitive profiles for agents in two scenarios: 68 actual contestants in the AnimalAI Olympics and 30 synthetic agents for O-PIAAGETS, an object permanence battery. We showcase the potential for capability-oriented evaluation. |
| title | Inferring Capabilities from Task Performance with Bayesian Triangulation |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2309.11975 |