When LLMs Meet Cunning Texts: A Fallacy Understanding Benchmark for Large Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913383344766976 |
|---|---|
| author | Li, Yinghui Zhou, Qingyu Luo, Yuanzhen Ma, Shirong Li, Yangning Zheng, Hai-Tao Hu, Xuming Yu, Philip S. |
| author_facet | Li, Yinghui Zhou, Qingyu Luo, Yuanzhen Ma, Shirong Li, Yangning Zheng, Hai-Tao Hu, Xuming Yu, Philip S. |
| contents | Recently, Large Language Models (LLMs) make remarkable evolutions in language understanding and generation. Following this, various benchmarks for measuring all kinds of capabilities of LLMs have sprung up. In this paper, we challenge the reasoning and understanding abilities of LLMs by proposing a FaLlacy Understanding Benchmark (FLUB) containing cunning texts that are easy for humans to understand but difficult for models to grasp. Specifically, the cunning texts that FLUB focuses on mainly consist of the tricky, humorous, and misleading texts collected from the real internet environment. And we design three tasks with increasing difficulty in the FLUB benchmark to evaluate the fallacy understanding ability of LLMs. Based on FLUB, we investigate the performance of multiple representative and advanced LLMs, reflecting our FLUB is challenging and worthy of more future study. Interesting discoveries and valuable insights are achieved in our extensive experiments and detailed analyses. We hope that our benchmark can encourage the community to improve LLMs' ability to understand fallacies. Our data and codes are available at https://github.com/THUKElab/FLUB. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_11100 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | When LLMs Meet Cunning Texts: A Fallacy Understanding Benchmark for Large Language Models Li, Yinghui Zhou, Qingyu Luo, Yuanzhen Ma, Shirong Li, Yangning Zheng, Hai-Tao Hu, Xuming Yu, Philip S. Computation and Language Recently, Large Language Models (LLMs) make remarkable evolutions in language understanding and generation. Following this, various benchmarks for measuring all kinds of capabilities of LLMs have sprung up. In this paper, we challenge the reasoning and understanding abilities of LLMs by proposing a FaLlacy Understanding Benchmark (FLUB) containing cunning texts that are easy for humans to understand but difficult for models to grasp. Specifically, the cunning texts that FLUB focuses on mainly consist of the tricky, humorous, and misleading texts collected from the real internet environment. And we design three tasks with increasing difficulty in the FLUB benchmark to evaluate the fallacy understanding ability of LLMs. Based on FLUB, we investigate the performance of multiple representative and advanced LLMs, reflecting our FLUB is challenging and worthy of more future study. Interesting discoveries and valuable insights are achieved in our extensive experiments and detailed analyses. We hope that our benchmark can encourage the community to improve LLMs' ability to understand fallacies. Our data and codes are available at https://github.com/THUKElab/FLUB. |
| title | When LLMs Meet Cunning Texts: A Fallacy Understanding Benchmark for Large Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2402.11100 |