Metacognitive Capabilities of LLMs: An Exploration in Mathematical Problem Solving
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909207301718016 |
|---|---|
| author | Didolkar, Aniket Goyal, Anirudh Ke, Nan Rosemary Guo, Siyuan Valko, Michal Lillicrap, Timothy Rezende, Danilo Bengio, Yoshua Mozer, Michael Arora, Sanjeev |
| author_facet | Didolkar, Aniket Goyal, Anirudh Ke, Nan Rosemary Guo, Siyuan Valko, Michal Lillicrap, Timothy Rezende, Danilo Bengio, Yoshua Mozer, Michael Arora, Sanjeev |
| contents | Metacognitive knowledge refers to humans' intuitive knowledge of their own thinking and reasoning processes. Today's best LLMs clearly possess some reasoning processes. The paper gives evidence that they also have metacognitive knowledge, including ability to name skills and procedures to apply given a task. We explore this primarily in context of math reasoning, developing a prompt-guided interaction procedure to get a powerful LLM to assign sensible skill labels to math questions, followed by having it perform semantic clustering to obtain coarser families of skill labels. These coarse skill labels look interpretable to humans.
To validate that these skill labels are meaningful and relevant to the LLM's reasoning processes we perform the following experiments. (a) We ask GPT-4 to assign skill labels to training questions in math datasets GSM8K and MATH. (b) When using an LLM to solve the test questions, we present it with the full list of skill labels and ask it to identify the skill needed. Then it is presented with randomly selected exemplar solved questions associated with that skill label. This improves accuracy on GSM8k and MATH for several strong LLMs, including code-assisted models. The methodology presented is domain-agnostic, even though this article applies it to math problems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_12205 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Metacognitive Capabilities of LLMs: An Exploration in Mathematical Problem Solving Didolkar, Aniket Goyal, Anirudh Ke, Nan Rosemary Guo, Siyuan Valko, Michal Lillicrap, Timothy Rezende, Danilo Bengio, Yoshua Mozer, Michael Arora, Sanjeev Artificial Intelligence Machine Learning Metacognitive knowledge refers to humans' intuitive knowledge of their own thinking and reasoning processes. Today's best LLMs clearly possess some reasoning processes. The paper gives evidence that they also have metacognitive knowledge, including ability to name skills and procedures to apply given a task. We explore this primarily in context of math reasoning, developing a prompt-guided interaction procedure to get a powerful LLM to assign sensible skill labels to math questions, followed by having it perform semantic clustering to obtain coarser families of skill labels. These coarse skill labels look interpretable to humans. To validate that these skill labels are meaningful and relevant to the LLM's reasoning processes we perform the following experiments. (a) We ask GPT-4 to assign skill labels to training questions in math datasets GSM8K and MATH. (b) When using an LLM to solve the test questions, we present it with the full list of skill labels and ask it to identify the skill needed. Then it is presented with randomly selected exemplar solved questions associated with that skill label. This improves accuracy on GSM8k and MATH for several strong LLMs, including code-assisted models. The methodology presented is domain-agnostic, even though this article applies it to math problems. |
| title | Metacognitive Capabilities of LLMs: An Exploration in Mathematical Problem Solving |
| topic | Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2405.12205 |