LogicAsker: Evaluating and Improving the Logical Reasoning Ability of Large Language Models
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909340946923520 |
|---|---|
| author | Wan, Yuxuan Wang, Wenxuan Yang, Yiliu Yuan, Youliang Huang, Jen-tse He, Pinjia Jiao, Wenxiang Lyu, Michael R. |
| author_facet | Wan, Yuxuan Wang, Wenxuan Yang, Yiliu Yuan, Youliang Huang, Jen-tse He, Pinjia Jiao, Wenxiang Lyu, Michael R. |
| contents | We introduce LogicAsker, a novel approach for evaluating and enhancing the logical reasoning capabilities of large language models (LLMs) such as ChatGPT and GPT-4. Despite LLMs' prowess in tasks like writing assistance, code generation, and machine translation, assessing their ability to reason has been challenging. Traditional evaluations often prioritize accuracy on downstream tasks over direct assessments of reasoning processes. LogicAsker addresses this gap by employing a set of atomic reasoning skills grounded in propositional and predicate logic to systematically examine and improve the reasoning prowess of LLMs. Our methodology reveals significant gaps in LLMs' learning of logical rules, with identified reasoning failures ranging from 29\% to 90\% across different models. Moreover, we leverage these findings to construct targeted demonstration examples and fine-tune data, notably enhancing logical reasoning in models like GPT-4o by up to 5\%. To our knowledge, this is the first effort to utilize test case outcomes to effectively refine LLMs' formal reasoning capabilities. We make our code, data, and results publicly available (https://github.com/yxwan123/LogicAsker) to facilitate further research and replication of our findings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_00757 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LogicAsker: Evaluating and Improving the Logical Reasoning Ability of Large Language Models Wan, Yuxuan Wang, Wenxuan Yang, Yiliu Yuan, Youliang Huang, Jen-tse He, Pinjia Jiao, Wenxiang Lyu, Michael R. Software Engineering Artificial Intelligence Computation and Language Logic in Computer Science We introduce LogicAsker, a novel approach for evaluating and enhancing the logical reasoning capabilities of large language models (LLMs) such as ChatGPT and GPT-4. Despite LLMs' prowess in tasks like writing assistance, code generation, and machine translation, assessing their ability to reason has been challenging. Traditional evaluations often prioritize accuracy on downstream tasks over direct assessments of reasoning processes. LogicAsker addresses this gap by employing a set of atomic reasoning skills grounded in propositional and predicate logic to systematically examine and improve the reasoning prowess of LLMs. Our methodology reveals significant gaps in LLMs' learning of logical rules, with identified reasoning failures ranging from 29\% to 90\% across different models. Moreover, we leverage these findings to construct targeted demonstration examples and fine-tune data, notably enhancing logical reasoning in models like GPT-4o by up to 5\%. To our knowledge, this is the first effort to utilize test case outcomes to effectively refine LLMs' formal reasoning capabilities. We make our code, data, and results publicly available (https://github.com/yxwan123/LogicAsker) to facilitate further research and replication of our findings. |
| title | LogicAsker: Evaluating and Improving the Logical Reasoning Ability of Large Language Models |
| topic | Software Engineering Artificial Intelligence Computation and Language Logic in Computer Science |
| url | https://arxiv.org/abs/2401.00757 |