First Heuristic Then Rational: Dynamic Use of Heuristics in Language Model Reasoning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916425054027776 |
|---|---|
| author | Aoki, Yoichi Kudo, Keito Kuribayashi, Tatsuki Sone, Shusaku Taniguchi, Masaya Sakaguchi, Keisuke Inui, Kentaro |
| author_facet | Aoki, Yoichi Kudo, Keito Kuribayashi, Tatsuki Sone, Shusaku Taniguchi, Masaya Sakaguchi, Keisuke Inui, Kentaro |
| contents | Multi-step reasoning instruction, such as chain-of-thought prompting, is widely adopted to explore better language models (LMs) performance. We report on the systematic strategy that LMs employ in such a multi-step reasoning process. Our controlled experiments reveal that LMs rely more heavily on heuristics, such as lexical overlap, in the earlier stages of reasoning, where more reasoning steps remain to reach a goal. Conversely, their reliance on heuristics decreases as LMs progress closer to the final answer through multiple reasoning steps. This suggests that LMs can backtrack only a limited number of future steps and dynamically combine heuristic strategies with rationale ones in tasks involving multi-step reasoning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_16078 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | First Heuristic Then Rational: Dynamic Use of Heuristics in Language Model Reasoning Aoki, Yoichi Kudo, Keito Kuribayashi, Tatsuki Sone, Shusaku Taniguchi, Masaya Sakaguchi, Keisuke Inui, Kentaro Computation and Language Multi-step reasoning instruction, such as chain-of-thought prompting, is widely adopted to explore better language models (LMs) performance. We report on the systematic strategy that LMs employ in such a multi-step reasoning process. Our controlled experiments reveal that LMs rely more heavily on heuristics, such as lexical overlap, in the earlier stages of reasoning, where more reasoning steps remain to reach a goal. Conversely, their reliance on heuristics decreases as LMs progress closer to the final answer through multiple reasoning steps. This suggests that LMs can backtrack only a limited number of future steps and dynamically combine heuristic strategies with rationale ones in tasks involving multi-step reasoning. |
| title | First Heuristic Then Rational: Dynamic Use of Heuristics in Language Model Reasoning |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2406.16078 |