Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2404.00205 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909155541909504 |
|---|---|
| author | Zhou, Ben Zhang, Hongming Chen, Sihao Yu, Dian Wang, Hongwei Peng, Baolin Roth, Dan Yu, Dong |
| author_facet | Zhou, Ben Zhang, Hongming Chen, Sihao Yu, Dian Wang, Hongwei Peng, Baolin Roth, Dan Yu, Dong |
| contents | Conceptual reasoning, the ability to reason in abstract and high-level perspectives, is key to generalization in human cognition. However, limited study has been done on large language models' capability to perform conceptual reasoning. In this work, we bridge this gap and propose a novel conceptualization framework that forces models to perform conceptual reasoning on abstract questions and generate solutions in a verifiable symbolic space. Using this framework as an analytical tool, we show that existing large language models fall short on conceptual reasoning, dropping 9% to 28% on various benchmarks compared to direct inference methods. We then discuss how models can improve since high-level abstract reasoning is key to unbiased and generalizable decision-making. We propose two techniques to add trustworthy induction signals by generating familiar questions with similar underlying reasoning paths and asking models to perform self-refinement. Experiments show that our proposed techniques improve models' conceptual reasoning performance by 8% to 11%, achieving a more robust reasoning system that relies less on inductive biases. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_00205 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Conceptual and Unbiased Reasoning in Language Models Zhou, Ben Zhang, Hongming Chen, Sihao Yu, Dian Wang, Hongwei Peng, Baolin Roth, Dan Yu, Dong Computation and Language Conceptual reasoning, the ability to reason in abstract and high-level perspectives, is key to generalization in human cognition. However, limited study has been done on large language models' capability to perform conceptual reasoning. In this work, we bridge this gap and propose a novel conceptualization framework that forces models to perform conceptual reasoning on abstract questions and generate solutions in a verifiable symbolic space. Using this framework as an analytical tool, we show that existing large language models fall short on conceptual reasoning, dropping 9% to 28% on various benchmarks compared to direct inference methods. We then discuss how models can improve since high-level abstract reasoning is key to unbiased and generalizable decision-making. We propose two techniques to add trustworthy induction signals by generating familiar questions with similar underlying reasoning paths and asking models to perform self-refinement. Experiments show that our proposed techniques improve models' conceptual reasoning performance by 8% to 11%, achieving a more robust reasoning system that relies less on inductive biases. |
| title | Conceptual and Unbiased Reasoning in Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2404.00205 |