Large Language Models Fall Short: Understanding Complex Relationships in Detective Narratives
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866911778817966080 |
|---|---|
| author | Zhao, Runcong Zhu, Qinglin Xu, Hainiu Li, Jiazheng Zhou, Yuxiang He, Yulan Gui, Lin |
| author_facet | Zhao, Runcong Zhu, Qinglin Xu, Hainiu Li, Jiazheng Zhou, Yuxiang He, Yulan Gui, Lin |
| contents | Existing datasets for narrative understanding often fail to represent the complexity and uncertainty of relationships in real-life social scenarios. To address this gap, we introduce a new benchmark, Conan, designed for extracting and analysing intricate character relation graphs from detective narratives. Specifically, we designed hierarchical relationship categories and manually extracted and annotated role-oriented relationships from the perspectives of various characters, incorporating both public relationships known to most characters and secret ones known to only a few. Our experiments with advanced Large Language Models (LLMs) like GPT-3.5, GPT-4, and Llama2 reveal their limitations in inferencing complex relationships and handling longer narratives. The combination of the Conan dataset and our pipeline strategy is geared towards understanding the ability of LLMs to comprehend nuanced relational dynamics in narrative contexts. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_11051 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Large Language Models Fall Short: Understanding Complex Relationships in Detective Narratives Zhao, Runcong Zhu, Qinglin Xu, Hainiu Li, Jiazheng Zhou, Yuxiang He, Yulan Gui, Lin Computation and Language Artificial Intelligence Existing datasets for narrative understanding often fail to represent the complexity and uncertainty of relationships in real-life social scenarios. To address this gap, we introduce a new benchmark, Conan, designed for extracting and analysing intricate character relation graphs from detective narratives. Specifically, we designed hierarchical relationship categories and manually extracted and annotated role-oriented relationships from the perspectives of various characters, incorporating both public relationships known to most characters and secret ones known to only a few. Our experiments with advanced Large Language Models (LLMs) like GPT-3.5, GPT-4, and Llama2 reveal their limitations in inferencing complex relationships and handling longer narratives. The combination of the Conan dataset and our pipeline strategy is geared towards understanding the ability of LLMs to comprehend nuanced relational dynamics in narrative contexts. |
| title | Large Language Models Fall Short: Understanding Complex Relationships in Detective Narratives |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2402.11051 |