It's Not Easy Being Wrong: Large Language Models Struggle with Process of Elimination Reasoning
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911910308347904 |
|---|---|
| author | Balepur, Nishant Palta, Shramay Rudinger, Rachel |
| author_facet | Balepur, Nishant Palta, Shramay Rudinger, Rachel |
| contents | Chain-of-thought (COT) prompting can help large language models (LLMs) reason toward correct answers, but its efficacy in reasoning toward incorrect answers is unexplored. This process of elimination (PoE), when used with COT, can enhance self-consistency, interpretability, and tasks such as medical diagnoses of exclusion. Thus, we propose PoE with COT, where LLMs must reason toward incorrect options on multiple-choice questions. We evaluate the ability of GPT-3.5, LLaMA-2, and Falcon to perform PoE with COT on a total of four commonsense and scientific reasoning datasets. We find that the strategy of PoE always underperforms the strategy of choosing the correct answer. The agreement of these strategies is also lower than the self-consistency of each strategy. To study these issues further, we conduct error analyses and give suggestions for future work. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2311_07532 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | It's Not Easy Being Wrong: Large Language Models Struggle with Process of Elimination Reasoning Balepur, Nishant Palta, Shramay Rudinger, Rachel Computation and Language Chain-of-thought (COT) prompting can help large language models (LLMs) reason toward correct answers, but its efficacy in reasoning toward incorrect answers is unexplored. This process of elimination (PoE), when used with COT, can enhance self-consistency, interpretability, and tasks such as medical diagnoses of exclusion. Thus, we propose PoE with COT, where LLMs must reason toward incorrect options on multiple-choice questions. We evaluate the ability of GPT-3.5, LLaMA-2, and Falcon to perform PoE with COT on a total of four commonsense and scientific reasoning datasets. We find that the strategy of PoE always underperforms the strategy of choosing the correct answer. The agreement of these strategies is also lower than the self-consistency of each strategy. To study these issues further, we conduct error analyses and give suggestions for future work. |
| title | It's Not Easy Being Wrong: Large Language Models Struggle with Process of Elimination Reasoning |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2311.07532 |