MAFALDA: A Benchmark and Comprehensive Study of Fallacy Detection and Classification
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929307912241152 |
|---|---|
| author | Helwe, Chadi Calamai, Tom Paris, Pierre-Henri Clavel, Chloé Suchanek, Fabian |
| author_facet | Helwe, Chadi Calamai, Tom Paris, Pierre-Henri Clavel, Chloé Suchanek, Fabian |
| contents | We introduce MAFALDA, a benchmark for fallacy classification that merges and unites previous fallacy datasets. It comes with a taxonomy that aligns, refines, and unifies existing classifications of fallacies. We further provide a manual annotation of a part of the dataset together with manual explanations for each annotation. We propose a new annotation scheme tailored for subjective NLP tasks, and a new evaluation method designed to handle subjectivity. We then evaluate several language models under a zero-shot learning setting and human performances on MAFALDA to assess their capability to detect and classify fallacies. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2311_09761 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | MAFALDA: A Benchmark and Comprehensive Study of Fallacy Detection and Classification Helwe, Chadi Calamai, Tom Paris, Pierre-Henri Clavel, Chloé Suchanek, Fabian Computation and Language Artificial Intelligence Machine Learning We introduce MAFALDA, a benchmark for fallacy classification that merges and unites previous fallacy datasets. It comes with a taxonomy that aligns, refines, and unifies existing classifications of fallacies. We further provide a manual annotation of a part of the dataset together with manual explanations for each annotation. We propose a new annotation scheme tailored for subjective NLP tasks, and a new evaluation method designed to handle subjectivity. We then evaluate several language models under a zero-shot learning setting and human performances on MAFALDA to assess their capability to detect and classify fallacies. |
| title | MAFALDA: A Benchmark and Comprehensive Study of Fallacy Detection and Classification |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2311.09761 |