Functional Benchmarks for Robust Evaluation of Reasoning Performance, and the Reasoning Gap
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917601101217792 |
|---|---|
| author | Srivastava, Saurabh B, Annarose M P V, Anto Menon, Shashank Sukumar, Ajay T, Adwaith Samod Philipose, Alan Prince, Stevin Thomas, Sooraj |
| author_facet | Srivastava, Saurabh B, Annarose M P V, Anto Menon, Shashank Sukumar, Ajay T, Adwaith Samod Philipose, Alan Prince, Stevin Thomas, Sooraj |
| contents | We propose a framework for robust evaluation of reasoning capabilities of language models, using functional variants of benchmarks. Models that solve a reasoning test should exhibit no difference in performance over the static version of a problem compared to a snapshot of the functional variant. We have rewritten the relevant fragment of the MATH benchmark into its functional variant MATH(), with functionalization of other benchmarks to follow. When evaluating current state-of-the-art models over snapshots of MATH(), we find a reasoning gap -- the percentage difference between the static and functional accuracies. We find reasoning gaps from 58.35% to 80.31% among the state-of-the-art closed and open weights models that perform well on static benchmarks, with the caveat that the gaps are likely to be smaller with more sophisticated prompting strategies. Here we show that models which anecdotally have good reasoning performance over real-world tasks, have quantifiable lower gaps, motivating the open problem of building "gap 0" models. Code for evaluation and new evaluation datasets, three MATH() snapshots, are publicly available at https://github.com/consequentai/fneval/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_19450 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Functional Benchmarks for Robust Evaluation of Reasoning Performance, and the Reasoning Gap Srivastava, Saurabh B, Annarose M P V, Anto Menon, Shashank Sukumar, Ajay T, Adwaith Samod Philipose, Alan Prince, Stevin Thomas, Sooraj Artificial Intelligence Computation and Language We propose a framework for robust evaluation of reasoning capabilities of language models, using functional variants of benchmarks. Models that solve a reasoning test should exhibit no difference in performance over the static version of a problem compared to a snapshot of the functional variant. We have rewritten the relevant fragment of the MATH benchmark into its functional variant MATH(), with functionalization of other benchmarks to follow. When evaluating current state-of-the-art models over snapshots of MATH(), we find a reasoning gap -- the percentage difference between the static and functional accuracies. We find reasoning gaps from 58.35% to 80.31% among the state-of-the-art closed and open weights models that perform well on static benchmarks, with the caveat that the gaps are likely to be smaller with more sophisticated prompting strategies. Here we show that models which anecdotally have good reasoning performance over real-world tasks, have quantifiable lower gaps, motivating the open problem of building "gap 0" models. Code for evaluation and new evaluation datasets, three MATH() snapshots, are publicly available at https://github.com/consequentai/fneval/. |
| title | Functional Benchmarks for Robust Evaluation of Reasoning Performance, and the Reasoning Gap |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2402.19450 |