Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2510.09970 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911203776790528 |
|---|---|
| author | Wang, Olivia Peiyu Bansal, Tashvi Bai, Ryan Chui, Emily M. Gilpin, Leilani H. |
| author_facet | Wang, Olivia Peiyu Bansal, Tashvi Bai, Ryan Chui, Emily M. Gilpin, Leilani H. |
| contents | Large Language Models (LLMs) suffer from critical reasoning gaps, including a tendency to hallucinate and poor accuracy in classifying logical fallacies. This limitation stems from their default System 1 processing, which is fast and intuitive, whereas reliable reasoning requires the deliberate, effortful System 2 approach (Kahneman, 2011; Li et al., 2025). Since full System 2 training is often prohibitively expensive, we explore a low-cost, instruction-based intervention to bridge this gap. Our methodology introduces a novel stepwise instruction dataset that decomposes fallacy classification into a series of atomic procedural steps (simple binary questions). We further augment this with a final verification step where models consult a relational knowledge graph of related fallacies. This procedural, rule-based intervention yields a significant improvement in LLM logical fallacy classification. Crucially, the approach also provides enhanced transparency into the LLMs' decision-making, highlighting a practical pathway for Neuro-symbolic architectures to address LLM reasoning deficits. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_09970 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Follow My Lead: Logical Fallacy Classification with Knowledge-Augmented LLMs Wang, Olivia Peiyu Bansal, Tashvi Bai, Ryan Chui, Emily M. Gilpin, Leilani H. Artificial Intelligence Large Language Models (LLMs) suffer from critical reasoning gaps, including a tendency to hallucinate and poor accuracy in classifying logical fallacies. This limitation stems from their default System 1 processing, which is fast and intuitive, whereas reliable reasoning requires the deliberate, effortful System 2 approach (Kahneman, 2011; Li et al., 2025). Since full System 2 training is often prohibitively expensive, we explore a low-cost, instruction-based intervention to bridge this gap. Our methodology introduces a novel stepwise instruction dataset that decomposes fallacy classification into a series of atomic procedural steps (simple binary questions). We further augment this with a final verification step where models consult a relational knowledge graph of related fallacies. This procedural, rule-based intervention yields a significant improvement in LLM logical fallacy classification. Crucially, the approach also provides enhanced transparency into the LLMs' decision-making, highlighting a practical pathway for Neuro-symbolic architectures to address LLM reasoning deficits. |
| title | Follow My Lead: Logical Fallacy Classification with Knowledge-Augmented LLMs |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2510.09970 |