Do Large Language Models Excel in Complex Logical Reasoning with Formal Language?
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912388049010688 |
|---|---|
| author | Jiang, Jin Wang, Jianing Yan, Yuchen Liu, Yang Zhu, Jianhua Zhang, Mengdi Cai, Xunliang Gao, Liangcai |
| author_facet | Jiang, Jin Wang, Jianing Yan, Yuchen Liu, Yang Zhu, Jianhua Zhang, Mengdi Cai, Xunliang Gao, Liangcai |
| contents | Large Language Models (LLMs) have been shown to achieve breakthrough performance on complex logical reasoning tasks. Nevertheless, most existing research focuses on employing formal language to guide LLMs to derive reliable reasoning paths, while systematic evaluations of these capabilities are still limited. In this paper, we aim to conduct a comprehensive evaluation of LLMs across various logical reasoning problems utilizing formal languages. From the perspective of three dimensions, i.e., spectrum of LLMs, taxonomy of tasks, and format of trajectories, our key findings are: 1) Thinking models significantly outperform Instruct models, especially when formal language is employed; 2) All LLMs exhibit limitations in inductive reasoning capability, irrespective of whether they use a formal language; 3) Data with PoT format achieves the best generalization performance across other languages. Additionally, we also curate the formal-relative training data to further enhance the small language models, and the experimental results indicate that a simple rejected fine-tuning method can better enable LLMs to generalize across formal languages and achieve the best overall performance. Our codes and reports are available at https://github.com/jiangjin1999/FormalEval. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_16998 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Do Large Language Models Excel in Complex Logical Reasoning with Formal Language? Jiang, Jin Wang, Jianing Yan, Yuchen Liu, Yang Zhu, Jianhua Zhang, Mengdi Cai, Xunliang Gao, Liangcai Computation and Language Artificial Intelligence Large Language Models (LLMs) have been shown to achieve breakthrough performance on complex logical reasoning tasks. Nevertheless, most existing research focuses on employing formal language to guide LLMs to derive reliable reasoning paths, while systematic evaluations of these capabilities are still limited. In this paper, we aim to conduct a comprehensive evaluation of LLMs across various logical reasoning problems utilizing formal languages. From the perspective of three dimensions, i.e., spectrum of LLMs, taxonomy of tasks, and format of trajectories, our key findings are: 1) Thinking models significantly outperform Instruct models, especially when formal language is employed; 2) All LLMs exhibit limitations in inductive reasoning capability, irrespective of whether they use a formal language; 3) Data with PoT format achieves the best generalization performance across other languages. Additionally, we also curate the formal-relative training data to further enhance the small language models, and the experimental results indicate that a simple rejected fine-tuning method can better enable LLMs to generalize across formal languages and achieve the best overall performance. Our codes and reports are available at https://github.com/jiangjin1999/FormalEval. |
| title | Do Large Language Models Excel in Complex Logical Reasoning with Formal Language? |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2505.16998 |