Dissecting Logical Reasoning in LLMs: A Fine-Grained Evaluation and Supervision Study
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911200080560128 |
|---|---|
| author | Zhou, Yujun Ye, Jiayi Ling, Zipeng Han, Yufei Huang, Yue Zhuang, Haomin Liang, Zhenwen Guo, Kehan Guo, Taicheng Wang, Xiangqi Zhang, Xiangliang |
| author_facet | Zhou, Yujun Ye, Jiayi Ling, Zipeng Han, Yufei Huang, Yue Zhuang, Haomin Liang, Zhenwen Guo, Kehan Guo, Taicheng Wang, Xiangqi Zhang, Xiangliang |
| contents | Logical reasoning is a core capability for large language models (LLMs), yet existing benchmarks that rely solely on final-answer accuracy fail to capture the quality of the reasoning process. To address this, we introduce FineLogic, a fine-grained evaluation framework that assesses logical reasoning across three dimensions: overall accuracy, stepwise soundness, and representation-level probing. Leveraging this framework, we conduct a comprehensive study on how different supervision formats in fine-tuning shape reasoning abilities. We fine-tune LLMs on four supervision styles: one in natural language and three symbolic variants. We find a key trade-off: natural language supervision excels at generalization to out-of-distribution and long-chain problems, whereas symbolic supervision is superior at instilling structurally sound, atomic reasoning steps. Furthermore, our probing analysis indicates that fine-tuning primarily refines the model's step-by-step generation process, rather than improving its ability to converge on an answer early. Together, our framework and analysis provide a more rigorous lens for evaluating and improving logical reasoning in LLMs. The code is available at https://github.com/YujunZhou/FineLogic. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_04810 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Dissecting Logical Reasoning in LLMs: A Fine-Grained Evaluation and Supervision Study Zhou, Yujun Ye, Jiayi Ling, Zipeng Han, Yufei Huang, Yue Zhuang, Haomin Liang, Zhenwen Guo, Kehan Guo, Taicheng Wang, Xiangqi Zhang, Xiangliang Computation and Language Artificial Intelligence Logic in Computer Science Logical reasoning is a core capability for large language models (LLMs), yet existing benchmarks that rely solely on final-answer accuracy fail to capture the quality of the reasoning process. To address this, we introduce FineLogic, a fine-grained evaluation framework that assesses logical reasoning across three dimensions: overall accuracy, stepwise soundness, and representation-level probing. Leveraging this framework, we conduct a comprehensive study on how different supervision formats in fine-tuning shape reasoning abilities. We fine-tune LLMs on four supervision styles: one in natural language and three symbolic variants. We find a key trade-off: natural language supervision excels at generalization to out-of-distribution and long-chain problems, whereas symbolic supervision is superior at instilling structurally sound, atomic reasoning steps. Furthermore, our probing analysis indicates that fine-tuning primarily refines the model's step-by-step generation process, rather than improving its ability to converge on an answer early. Together, our framework and analysis provide a more rigorous lens for evaluating and improving logical reasoning in LLMs. The code is available at https://github.com/YujunZhou/FineLogic. |
| title | Dissecting Logical Reasoning in LLMs: A Fine-Grained Evaluation and Supervision Study |
| topic | Computation and Language Artificial Intelligence Logic in Computer Science |
| url | https://arxiv.org/abs/2506.04810 |