Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2410.01920 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929735679868928 |
|---|---|
| author | Feng, Shengyu Kong, Xiang Ma, Shuang Zhang, Aonan Yin, Dong Wang, Chong Pang, Ruoming Yang, Yiming |
| author_facet | Feng, Shengyu Kong, Xiang Ma, Shuang Zhang, Aonan Yin, Dong Wang, Chong Pang, Ruoming Yang, Yiming |
| contents | Augmenting the multi-step reasoning abilities of Large Language Models (LLMs) has been a persistent challenge. Recently, verification has shown promise in improving solution consistency by evaluating generated outputs. However, current verification approaches suffer from sampling inefficiencies, requiring a large number of samples to achieve satisfactory performance. Additionally, training an effective verifier often depends on extensive process supervision, which is costly to acquire. In this paper, we address these limitations by introducing a novel verification method based on Twisted Sequential Monte Carlo (TSMC). TSMC sequentially refines its sampling effort to focus exploration on promising candidates, resulting in more efficient generation of high-quality solutions. We apply TSMC to LLMs by estimating the expected future rewards at partial solutions. This approach results in a more straightforward training target that eliminates the need for step-wise human annotations. We empirically demonstrate the advantages of our method across multiple math benchmarks, and also validate our theoretical analysis of both our approach and existing verification methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_01920 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Step-by-Step Reasoning for Math Problems via Twisted Sequential Monte Carlo Feng, Shengyu Kong, Xiang Ma, Shuang Zhang, Aonan Yin, Dong Wang, Chong Pang, Ruoming Yang, Yiming Machine Learning Augmenting the multi-step reasoning abilities of Large Language Models (LLMs) has been a persistent challenge. Recently, verification has shown promise in improving solution consistency by evaluating generated outputs. However, current verification approaches suffer from sampling inefficiencies, requiring a large number of samples to achieve satisfactory performance. Additionally, training an effective verifier often depends on extensive process supervision, which is costly to acquire. In this paper, we address these limitations by introducing a novel verification method based on Twisted Sequential Monte Carlo (TSMC). TSMC sequentially refines its sampling effort to focus exploration on promising candidates, resulting in more efficient generation of high-quality solutions. We apply TSMC to LLMs by estimating the expected future rewards at partial solutions. This approach results in a more straightforward training target that eliminates the need for step-wise human annotations. We empirically demonstrate the advantages of our method across multiple math benchmarks, and also validate our theoretical analysis of both our approach and existing verification methods. |
| title | Step-by-Step Reasoning for Math Problems via Twisted Sequential Monte Carlo |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2410.01920 |