The CLRS-Text Algorithmic Reasoning Language Benchmark
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866916278088761344 |
|---|---|
| author | Markeeva, Larisa McLeish, Sean Ibarz, Borja Bounsi, Wilfried Kozlova, Olga Vitvitskyi, Alex Blundell, Charles Goldstein, Tom Schwarzschild, Avi Veličković, Petar |
| author_facet | Markeeva, Larisa McLeish, Sean Ibarz, Borja Bounsi, Wilfried Kozlova, Olga Vitvitskyi, Alex Blundell, Charles Goldstein, Tom Schwarzschild, Avi Veličković, Petar |
| contents | Eliciting reasoning capabilities from language models (LMs) is a critical direction on the path towards building intelligent systems. Most recent studies dedicated to reasoning focus on out-of-distribution performance on procedurally-generated synthetic benchmarks, bespoke-built to evaluate specific skills only. This trend makes results hard to transfer across publications, slowing down progress. Three years ago, a similar issue was identified and rectified in the field of neural algorithmic reasoning, with the advent of the CLRS benchmark. CLRS is a dataset generator comprising graph execution traces of classical algorithms from the Introduction to Algorithms textbook. Inspired by this, we propose CLRS-Text -- a textual version of these algorithmic traces. Out of the box, CLRS-Text is capable of procedurally generating trace data for thirty diverse, challenging algorithmic tasks across any desirable input distribution, while offering a standard pipeline in which any additional algorithmic tasks may be created in the benchmark. We fine-tune and evaluate various LMs as generalist executors on this benchmark, validating prior work and revealing a novel, interesting challenge for the LM reasoning community. Our code is available at https://github.com/google-deepmind/clrs/tree/master/clrs/_src/clrs_text. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_04229 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | The CLRS-Text Algorithmic Reasoning Language Benchmark Markeeva, Larisa McLeish, Sean Ibarz, Borja Bounsi, Wilfried Kozlova, Olga Vitvitskyi, Alex Blundell, Charles Goldstein, Tom Schwarzschild, Avi Veličković, Petar Machine Learning Artificial Intelligence Computation and Language Data Structures and Algorithms Eliciting reasoning capabilities from language models (LMs) is a critical direction on the path towards building intelligent systems. Most recent studies dedicated to reasoning focus on out-of-distribution performance on procedurally-generated synthetic benchmarks, bespoke-built to evaluate specific skills only. This trend makes results hard to transfer across publications, slowing down progress. Three years ago, a similar issue was identified and rectified in the field of neural algorithmic reasoning, with the advent of the CLRS benchmark. CLRS is a dataset generator comprising graph execution traces of classical algorithms from the Introduction to Algorithms textbook. Inspired by this, we propose CLRS-Text -- a textual version of these algorithmic traces. Out of the box, CLRS-Text is capable of procedurally generating trace data for thirty diverse, challenging algorithmic tasks across any desirable input distribution, while offering a standard pipeline in which any additional algorithmic tasks may be created in the benchmark. We fine-tune and evaluate various LMs as generalist executors on this benchmark, validating prior work and revealing a novel, interesting challenge for the LM reasoning community. Our code is available at https://github.com/google-deepmind/clrs/tree/master/clrs/_src/clrs_text. |
| title | The CLRS-Text Algorithmic Reasoning Language Benchmark |
| topic | Machine Learning Artificial Intelligence Computation and Language Data Structures and Algorithms |
| url | https://arxiv.org/abs/2406.04229 |