Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2411.02435 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910685107060736 |
|---|---|
| author | Leng, Xinyi Liang, Jason Mauro, Jack Wang, Xu Bertozzi, Andrea L. Chapman, James Lin, Junyuan Chen, Bohan Ye, Chenchen Daniel, Temple Brantingham, P. Jeffrey |
| author_facet | Leng, Xinyi Liang, Jason Mauro, Jack Wang, Xu Bertozzi, Andrea L. Chapman, James Lin, Junyuan Chen, Bohan Ye, Chenchen Daniel, Temple Brantingham, P. Jeffrey |
| contents | Narrative data spans all disciplines and provides a coherent model of the world to the reader or viewer. Recent advancement in machine learning and Large Language Models (LLMs) have enable great strides in analyzing natural language. However, Large language models (LLMs) still struggle with complex narrative arcs as well as narratives containing conflicting information. Recent work indicates LLMs augmented with external knowledge bases can improve the accuracy and interpretability of the resulting models. In this work, we analyze the effectiveness of applying knowledge graphs (KGs) in understanding true-crime podcast data from both classical Natural Language Processing (NLP) and LLM approaches. We directly compare KG-augmented LLMs (KGLLMs) with classical methods for KG construction, topic modeling, and sentiment analysis. Additionally, the KGLLM allows us to query the knowledge base in natural language and test its ability to factually answer questions. We examine the robustness of the model to adversarial prompting in order to test the model's ability to deal with conflicting information. Finally, we apply classical methods to understand more subtle aspects of the text such as the use of hearsay and sentiment in narrative construction and propose future directions. Our results indicate that KGLLMs outperform LLMs on a variety of metrics, are more robust to adversarial prompts, and are more capable of summarizing the text into topics. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_02435 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Narrative Analysis of True Crime Podcasts With Knowledge Graph-Augmented Large Language Models Leng, Xinyi Liang, Jason Mauro, Jack Wang, Xu Bertozzi, Andrea L. Chapman, James Lin, Junyuan Chen, Bohan Ye, Chenchen Daniel, Temple Brantingham, P. Jeffrey Computation and Language Machine Learning Narrative data spans all disciplines and provides a coherent model of the world to the reader or viewer. Recent advancement in machine learning and Large Language Models (LLMs) have enable great strides in analyzing natural language. However, Large language models (LLMs) still struggle with complex narrative arcs as well as narratives containing conflicting information. Recent work indicates LLMs augmented with external knowledge bases can improve the accuracy and interpretability of the resulting models. In this work, we analyze the effectiveness of applying knowledge graphs (KGs) in understanding true-crime podcast data from both classical Natural Language Processing (NLP) and LLM approaches. We directly compare KG-augmented LLMs (KGLLMs) with classical methods for KG construction, topic modeling, and sentiment analysis. Additionally, the KGLLM allows us to query the knowledge base in natural language and test its ability to factually answer questions. We examine the robustness of the model to adversarial prompting in order to test the model's ability to deal with conflicting information. Finally, we apply classical methods to understand more subtle aspects of the text such as the use of hearsay and sentiment in narrative construction and propose future directions. Our results indicate that KGLLMs outperform LLMs on a variety of metrics, are more robust to adversarial prompts, and are more capable of summarizing the text into topics. |
| title | Narrative Analysis of True Crime Podcasts With Knowledge Graph-Augmented Large Language Models |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2411.02435 |