Library Learning Doesn't: The Curious Case of the Single-Use "Library"
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914992350035968 |
|---|---|
| author | Berlot-Attwell, Ian Rudzicz, Frank Si, Xujie |
| author_facet | Berlot-Attwell, Ian Rudzicz, Frank Si, Xujie |
| contents | Advances in Large Language Models (LLMs) have spurred a wave of LLM library learning systems for mathematical reasoning. These systems aim to learn a reusable library of tools, such as formal Isabelle lemmas or Python programs that are tailored to a family of tasks. Many of these systems are inspired by the human structuring of knowledge into reusable and extendable concepts, but do current methods actually learn reusable libraries of tools?
We study two library learning systems for mathematics which both reported increased accuracy: LEGO-Prover and TroVE. We find that function reuse is extremely infrequent on miniF2F and MATH. Our followup ablation experiments suggest that, rather than reuse, self-correction and self-consistency are the primary drivers of the observed performance gains. Our code and data are available at https://github.com/ikb-a/curious-case |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_20274 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Library Learning Doesn't: The Curious Case of the Single-Use "Library" Berlot-Attwell, Ian Rudzicz, Frank Si, Xujie Machine Learning Computation and Language Symbolic Computation Advances in Large Language Models (LLMs) have spurred a wave of LLM library learning systems for mathematical reasoning. These systems aim to learn a reusable library of tools, such as formal Isabelle lemmas or Python programs that are tailored to a family of tasks. Many of these systems are inspired by the human structuring of knowledge into reusable and extendable concepts, but do current methods actually learn reusable libraries of tools? We study two library learning systems for mathematics which both reported increased accuracy: LEGO-Prover and TroVE. We find that function reuse is extremely infrequent on miniF2F and MATH. Our followup ablation experiments suggest that, rather than reuse, self-correction and self-consistency are the primary drivers of the observed performance gains. Our code and data are available at https://github.com/ikb-a/curious-case |
| title | Library Learning Doesn't: The Curious Case of the Single-Use "Library" |
| topic | Machine Learning Computation and Language Symbolic Computation |
| url | https://arxiv.org/abs/2410.20274 |