Towards Better Chain-of-Thought: A Reflection on Effectiveness and Faithfulness
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915314370871296 |
|---|---|
| author | Li, Jiachun Cao, Pengfei Chen, Yubo Xu, Jiexin Li, Huaijun Jiang, Xiaojian Liu, Kang Zhao, Jun |
| author_facet | Li, Jiachun Cao, Pengfei Chen, Yubo Xu, Jiexin Li, Huaijun Jiang, Xiaojian Liu, Kang Zhao, Jun |
| contents | Chain-of-thought (CoT) prompting demonstrates varying performance under different reasoning tasks. Previous work attempts to evaluate it but falls short in providing an in-depth analysis of patterns that influence the CoT. In this paper, we study the CoT performance from the perspective of effectiveness and faithfulness. For the former, we identify key factors that influence CoT effectiveness on performance improvement, including problem difficulty, information gain, and information flow. For the latter, we interpret the unfaithful CoT issue by conducting a joint analysis of the information interaction among the question, CoT, and answer. The result demonstrates that, when the LLM predicts answers, it can recall correct information missing in the CoT from the question, leading to the problem. Finally, we propose a novel algorithm to mitigate this issue, in which we recall extra information from the question to enhance the CoT generation and evaluate CoTs based on their information gain. Extensive experiments demonstrate that our approach enhances both the faithfulness and effectiveness of CoT. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_18915 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Towards Better Chain-of-Thought: A Reflection on Effectiveness and Faithfulness Li, Jiachun Cao, Pengfei Chen, Yubo Xu, Jiexin Li, Huaijun Jiang, Xiaojian Liu, Kang Zhao, Jun Computation and Language Artificial Intelligence Chain-of-thought (CoT) prompting demonstrates varying performance under different reasoning tasks. Previous work attempts to evaluate it but falls short in providing an in-depth analysis of patterns that influence the CoT. In this paper, we study the CoT performance from the perspective of effectiveness and faithfulness. For the former, we identify key factors that influence CoT effectiveness on performance improvement, including problem difficulty, information gain, and information flow. For the latter, we interpret the unfaithful CoT issue by conducting a joint analysis of the information interaction among the question, CoT, and answer. The result demonstrates that, when the LLM predicts answers, it can recall correct information missing in the CoT from the question, leading to the problem. Finally, we propose a novel algorithm to mitigate this issue, in which we recall extra information from the question to enhance the CoT generation and evaluate CoTs based on their information gain. Extensive experiments demonstrate that our approach enhances both the faithfulness and effectiveness of CoT. |
| title | Towards Better Chain-of-Thought: A Reflection on Effectiveness and Faithfulness |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2405.18915 |