Escape Sky-high Cost: Early-stopping Self-Consistency for Multi-step Reasoning
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910302721802240 |
|---|---|
| author | Li, Yiwei Yuan, Peiwen Feng, Shaoxiong Pan, Boyuan Wang, Xinglin Sun, Bin Wang, Heda Li, Kan |
| author_facet | Li, Yiwei Yuan, Peiwen Feng, Shaoxiong Pan, Boyuan Wang, Xinglin Sun, Bin Wang, Heda Li, Kan |
| contents | Self-consistency (SC) has been a widely used decoding strategy for chain-of-thought reasoning. Despite bringing significant performance improvements across a variety of multi-step reasoning tasks, it is a high-cost method that requires multiple sampling with the preset size. In this paper, we propose a simple and scalable sampling process, \textbf{E}arly-Stopping \textbf{S}elf-\textbf{C}onsistency (ESC), to greatly reduce the cost of SC without sacrificing performance. On this basis, one control scheme for ESC is further derivated to dynamically choose the performance-cost balance for different tasks and models. To demonstrate ESC's effectiveness, we conducted extensive experiments on three popular categories of reasoning tasks: arithmetic, commonsense and symbolic reasoning over language models with varying scales. The empirical results show that ESC reduces the average number of sampling of chain-of-thought reasoning by a significant margin on six benchmarks, including MATH (-33.8%), GSM8K (-80.1%), StrategyQA (-76.8%), CommonsenseQA (-78.5%), Coin Flip (-84.2%) and Last Letters (-67.4%), while attaining comparable performances. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_10480 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Escape Sky-high Cost: Early-stopping Self-Consistency for Multi-step Reasoning Li, Yiwei Yuan, Peiwen Feng, Shaoxiong Pan, Boyuan Wang, Xinglin Sun, Bin Wang, Heda Li, Kan Computation and Language Artificial Intelligence Self-consistency (SC) has been a widely used decoding strategy for chain-of-thought reasoning. Despite bringing significant performance improvements across a variety of multi-step reasoning tasks, it is a high-cost method that requires multiple sampling with the preset size. In this paper, we propose a simple and scalable sampling process, \textbf{E}arly-Stopping \textbf{S}elf-\textbf{C}onsistency (ESC), to greatly reduce the cost of SC without sacrificing performance. On this basis, one control scheme for ESC is further derivated to dynamically choose the performance-cost balance for different tasks and models. To demonstrate ESC's effectiveness, we conducted extensive experiments on three popular categories of reasoning tasks: arithmetic, commonsense and symbolic reasoning over language models with varying scales. The empirical results show that ESC reduces the average number of sampling of chain-of-thought reasoning by a significant margin on six benchmarks, including MATH (-33.8%), GSM8K (-80.1%), StrategyQA (-76.8%), CommonsenseQA (-78.5%), Coin Flip (-84.2%) and Last Letters (-67.4%), while attaining comparable performances. |
| title | Escape Sky-high Cost: Early-stopping Self-Consistency for Multi-step Reasoning |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2401.10480 |