Semantic Self-Consistency: Enhancing Language Model Reasoning via Semantic Weighting
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909468114026496 |
|---|---|
| author | Knappe, Tim Li, Ryan Chauhan, Ayush Chhua, Kaylee Zhu, Kevin O'Brien, Sean |
| author_facet | Knappe, Tim Li, Ryan Chauhan, Ayush Chhua, Kaylee Zhu, Kevin O'Brien, Sean |
| contents | While large language models (LLMs) have rapidly improved their performance on a broad number of tasks, they still often fall short on reasoning tasks. As LLMs become more integrated in diverse real-world tasks, advancing their reasoning capabilities is crucial to their effectiveness in nuanced, complex problems. Wang et al.'s self-consistency framework reveals that sampling multiple rationales before taking a majority vote reliably improves model performance across various closed-answer reasoning tasks. Standard methods based on this framework aggregate the final decisions of these rationales but fail to utilize the semantic information detailed in the step-by-step reasoning paths. Our work introduces semantic self-consistency, enhancing this approach by incorporating and analyzing both the reasoning paths of these rationales in addition to their final decisions before taking a majority vote. These methods not only improve the reliability of reasoning paths but also cause more robust performance on complex reasoning tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_07839 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Semantic Self-Consistency: Enhancing Language Model Reasoning via Semantic Weighting Knappe, Tim Li, Ryan Chauhan, Ayush Chhua, Kaylee Zhu, Kevin O'Brien, Sean Computation and Language While large language models (LLMs) have rapidly improved their performance on a broad number of tasks, they still often fall short on reasoning tasks. As LLMs become more integrated in diverse real-world tasks, advancing their reasoning capabilities is crucial to their effectiveness in nuanced, complex problems. Wang et al.'s self-consistency framework reveals that sampling multiple rationales before taking a majority vote reliably improves model performance across various closed-answer reasoning tasks. Standard methods based on this framework aggregate the final decisions of these rationales but fail to utilize the semantic information detailed in the step-by-step reasoning paths. Our work introduces semantic self-consistency, enhancing this approach by incorporating and analyzing both the reasoning paths of these rationales in addition to their final decisions before taking a majority vote. These methods not only improve the reliability of reasoning paths but also cause more robust performance on complex reasoning tasks. |
| title | Semantic Self-Consistency: Enhancing Language Model Reasoning via Semantic Weighting |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2410.07839 |