Self-Explore: Enhancing Mathematical Reasoning in Language Models with Fine-grained Rewards
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910630026412032 |
|---|---|
| author | Hwang, Hyeonbin Kim, Doyoung Kim, Seungone Ye, Seonghyeon Seo, Minjoon |
| author_facet | Hwang, Hyeonbin Kim, Doyoung Kim, Seungone Ye, Seonghyeon Seo, Minjoon |
| contents | Training on large amounts of rationales (i.e., CoT Fine-tuning) is effective at improving the reasoning capabilities of large language models (LLMs). However, acquiring human-authored rationales or augmenting rationales from proprietary models is costly and not scalable. In this paper, we study the problem of whether LLMs could self-improve their reasoning capabilities. To this end, we propose Self-Explore, where the LLM is tasked to explore the first wrong step (i.e., the first pit) within the rationale and use such signals as fine-grained rewards for further improvement. On the GSM8K and MATH test set, Self-Explore achieves 11.57% and 2.89% improvement on average across three LLMs compared to supervised fine-tuning (SFT). Our code is available at https://github.com/hbin0701/Self-Explore. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_10346 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Self-Explore: Enhancing Mathematical Reasoning in Language Models with Fine-grained Rewards Hwang, Hyeonbin Kim, Doyoung Kim, Seungone Ye, Seonghyeon Seo, Minjoon Computation and Language Training on large amounts of rationales (i.e., CoT Fine-tuning) is effective at improving the reasoning capabilities of large language models (LLMs). However, acquiring human-authored rationales or augmenting rationales from proprietary models is costly and not scalable. In this paper, we study the problem of whether LLMs could self-improve their reasoning capabilities. To this end, we propose Self-Explore, where the LLM is tasked to explore the first wrong step (i.e., the first pit) within the rationale and use such signals as fine-grained rewards for further improvement. On the GSM8K and MATH test set, Self-Explore achieves 11.57% and 2.89% improvement on average across three LLMs compared to supervised fine-tuning (SFT). Our code is available at https://github.com/hbin0701/Self-Explore. |
| title | Self-Explore: Enhancing Mathematical Reasoning in Language Models with Fine-grained Rewards |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2404.10346 |