Unveiling Narrative Reasoning Limits of Large Language Models with Trope in Movie Synopses
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916405953167360 |
|---|---|
| author | Su, Hung-Ting Hsu, Ya-Ching Lin, Xudong Shi, Xiang-Qian Niu, Yulei Hsu, Han-Yuan Lee, Hung-yi Hsu, Winston H. |
| author_facet | Su, Hung-Ting Hsu, Ya-Ching Lin, Xudong Shi, Xiang-Qian Niu, Yulei Hsu, Han-Yuan Lee, Hung-yi Hsu, Winston H. |
| contents | Large language models (LLMs) equipped with chain-of-thoughts (CoT) prompting have shown significant multi-step reasoning capabilities in factual content like mathematics, commonsense, and logic. However, their performance in narrative reasoning, which demands greater abstraction capabilities, remains unexplored. This study utilizes tropes in movie synopses to assess the abstract reasoning abilities of state-of-the-art LLMs and uncovers their low performance. We introduce a trope-wise querying approach to address these challenges and boost the F1 score by 11.8 points. Moreover, while prior studies suggest that CoT enhances multi-step reasoning, this study shows CoT can cause hallucinations in narrative content, reducing GPT-4's performance. We also introduce an Adversarial Injection method to embed trope-related text tokens into movie synopses without explicit tropes, revealing CoT's heightened sensitivity to such injections. Our comprehensive analysis provides insights for future research directions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_14324 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Unveiling Narrative Reasoning Limits of Large Language Models with Trope in Movie Synopses Su, Hung-Ting Hsu, Ya-Ching Lin, Xudong Shi, Xiang-Qian Niu, Yulei Hsu, Han-Yuan Lee, Hung-yi Hsu, Winston H. Computation and Language Artificial Intelligence Machine Learning Large language models (LLMs) equipped with chain-of-thoughts (CoT) prompting have shown significant multi-step reasoning capabilities in factual content like mathematics, commonsense, and logic. However, their performance in narrative reasoning, which demands greater abstraction capabilities, remains unexplored. This study utilizes tropes in movie synopses to assess the abstract reasoning abilities of state-of-the-art LLMs and uncovers their low performance. We introduce a trope-wise querying approach to address these challenges and boost the F1 score by 11.8 points. Moreover, while prior studies suggest that CoT enhances multi-step reasoning, this study shows CoT can cause hallucinations in narrative content, reducing GPT-4's performance. We also introduce an Adversarial Injection method to embed trope-related text tokens into movie synopses without explicit tropes, revealing CoT's heightened sensitivity to such injections. Our comprehensive analysis provides insights for future research directions. |
| title | Unveiling Narrative Reasoning Limits of Large Language Models with Trope in Movie Synopses |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2409.14324 |