Two Failures of Self-Consistency in the Multi-Step Reasoning of LLMs
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2023
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911773185015808 |
|---|---|
| author | Chen, Angelica Phang, Jason Parrish, Alicia Padmakumar, Vishakh Zhao, Chen Bowman, Samuel R. Cho, Kyunghyun |
| author_facet | Chen, Angelica Phang, Jason Parrish, Alicia Padmakumar, Vishakh Zhao, Chen Bowman, Samuel R. Cho, Kyunghyun |
| contents | Large language models (LLMs) have achieved widespread success on a variety of in-context few-shot tasks, but this success is typically evaluated via correctness rather than consistency. We argue that self-consistency is an important criteria for valid multi-step reasoning in tasks where the solution is composed of the answers to multiple sub-steps. We propose two types of self-consistency that are particularly important for multi-step reasoning -- hypothetical consistency (a model's ability to predict what its output would be in a hypothetical other context) and compositional consistency (consistency of a model's final outputs when intermediate sub-steps are replaced with the model's outputs for those steps). We demonstrate that multiple variants of the GPT-3/-4 models exhibit poor consistency rates across both types of consistency on a variety of tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2305_14279 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Two Failures of Self-Consistency in the Multi-Step Reasoning of LLMs Chen, Angelica Phang, Jason Parrish, Alicia Padmakumar, Vishakh Zhao, Chen Bowman, Samuel R. Cho, Kyunghyun Computation and Language Large language models (LLMs) have achieved widespread success on a variety of in-context few-shot tasks, but this success is typically evaluated via correctness rather than consistency. We argue that self-consistency is an important criteria for valid multi-step reasoning in tasks where the solution is composed of the answers to multiple sub-steps. We propose two types of self-consistency that are particularly important for multi-step reasoning -- hypothetical consistency (a model's ability to predict what its output would be in a hypothetical other context) and compositional consistency (consistency of a model's final outputs when intermediate sub-steps are replaced with the model's outputs for those steps). We demonstrate that multiple variants of the GPT-3/-4 models exhibit poor consistency rates across both types of consistency on a variety of tasks. |
| title | Two Failures of Self-Consistency in the Multi-Step Reasoning of LLMs |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2305.14279 |