Assessing the Impact of Conspiracy Theories Using Large Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912149898526720 |
|---|---|
| author | Jiang, Bohan Li, Dawei Tan, Zhen Zhou, Xinyi Rao, Ashwin Lerman, Kristina Bernard, H. Russell Liu, Huan |
| author_facet | Jiang, Bohan Li, Dawei Tan, Zhen Zhou, Xinyi Rao, Ashwin Lerman, Kristina Bernard, H. Russell Liu, Huan |
| contents | Measuring the relative impact of CTs is important for prioritizing responses and allocating resources effectively, especially during crises. However, assessing the actual impact of CTs on the public poses unique challenges. It requires not only the collection of CT-specific knowledge but also diverse information from social, psychological, and cultural dimensions. Recent advancements in large language models (LLMs) suggest their potential utility in this context, not only due to their extensive knowledge from large training corpora but also because they can be harnessed for complex reasoning. In this work, we develop datasets of popular CTs with human-annotated impacts. Borrowing insights from human impact assessment processes, we then design tailored strategies to leverage LLMs for performing human-like CT impact assessments. Through rigorous experiments, we textit{discover that an impact assessment mode using multi-step reasoning to analyze more CT-related evidence critically produces accurate results; and most LLMs demonstrate strong bias, such as assigning higher impacts to CTs presented earlier in the prompt, while generating less accurate impact assessments for emotionally charged and verbose CTs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_07019 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Assessing the Impact of Conspiracy Theories Using Large Language Models Jiang, Bohan Li, Dawei Tan, Zhen Zhou, Xinyi Rao, Ashwin Lerman, Kristina Bernard, H. Russell Liu, Huan Computation and Language Computers and Society Measuring the relative impact of CTs is important for prioritizing responses and allocating resources effectively, especially during crises. However, assessing the actual impact of CTs on the public poses unique challenges. It requires not only the collection of CT-specific knowledge but also diverse information from social, psychological, and cultural dimensions. Recent advancements in large language models (LLMs) suggest their potential utility in this context, not only due to their extensive knowledge from large training corpora but also because they can be harnessed for complex reasoning. In this work, we develop datasets of popular CTs with human-annotated impacts. Borrowing insights from human impact assessment processes, we then design tailored strategies to leverage LLMs for performing human-like CT impact assessments. Through rigorous experiments, we textit{discover that an impact assessment mode using multi-step reasoning to analyze more CT-related evidence critically produces accurate results; and most LLMs demonstrate strong bias, such as assigning higher impacts to CTs presented earlier in the prompt, while generating less accurate impact assessments for emotionally charged and verbose CTs. |
| title | Assessing the Impact of Conspiracy Theories Using Large Language Models |
| topic | Computation and Language Computers and Society |
| url | https://arxiv.org/abs/2412.07019 |