Benchmarking Defeasible Reasoning with Large Language Models -- Initial Experiments and Future Directions
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914974577721344 |
|---|---|
| author | Tachmazidis, Ilias Batsakis, Sotiris Antoniou, Grigoris |
| author_facet | Tachmazidis, Ilias Batsakis, Sotiris Antoniou, Grigoris |
| contents | Large Language Models (LLMs) have gained prominence in the AI landscape due to their exceptional performance. Thus, it is essential to gain a better understanding of their capabilities and limitations, among others in terms of nonmonotonic reasoning. This paper proposes a benchmark that corresponds to various defeasible rule-based reasoning patterns. We modified an existing benchmark for defeasible logic reasoners by translating defeasible rules into text suitable for LLMs. We conducted preliminary experiments on nonmonotonic rule-based reasoning using ChatGPT and compared it with reasoning patterns defined by defeasible logic. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_12509 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Benchmarking Defeasible Reasoning with Large Language Models -- Initial Experiments and Future Directions Tachmazidis, Ilias Batsakis, Sotiris Antoniou, Grigoris Artificial Intelligence Large Language Models (LLMs) have gained prominence in the AI landscape due to their exceptional performance. Thus, it is essential to gain a better understanding of their capabilities and limitations, among others in terms of nonmonotonic reasoning. This paper proposes a benchmark that corresponds to various defeasible rule-based reasoning patterns. We modified an existing benchmark for defeasible logic reasoners by translating defeasible rules into text suitable for LLMs. We conducted preliminary experiments on nonmonotonic rule-based reasoning using ChatGPT and compared it with reasoning patterns defined by defeasible logic. |
| title | Benchmarking Defeasible Reasoning with Large Language Models -- Initial Experiments and Future Directions |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2410.12509 |