How LLMs Fail to Support Fact-Checking
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866918148444258304 |
|---|---|
| author | Proma, Adiba Mahbub Pate, Neeley Druckman, James Ghoshal, Gourab He, Hangfeng Hoque, Ehsan |
| author_facet | Proma, Adiba Mahbub Pate, Neeley Druckman, James Ghoshal, Gourab He, Hangfeng Hoque, Ehsan |
| contents | While Large Language Models (LLMs) can amplify online misinformation, they also show promise in tackling misinformation. In this paper, we empirically study the capabilities of three LLMs -- ChatGPT, Gemini, and Claude -- in countering political misinformation. We implement a two-step, chain-of-thought prompting approach, where models first identify credible sources for a given claim and then generate persuasive responses. Our findings suggest that models struggle to ground their responses in real news sources, and tend to prefer citing left-leaning sources. We also observe varying degrees of response diversity among models. Our findings highlight concerns about using LLMs for fact-checking through only prompt-engineering, emphasizing the need for more robust guardrails. Our results have implications for both researchers and non-technical users. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_01902 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | How LLMs Fail to Support Fact-Checking Proma, Adiba Mahbub Pate, Neeley Druckman, James Ghoshal, Gourab He, Hangfeng Hoque, Ehsan Computation and Language Artificial Intelligence While Large Language Models (LLMs) can amplify online misinformation, they also show promise in tackling misinformation. In this paper, we empirically study the capabilities of three LLMs -- ChatGPT, Gemini, and Claude -- in countering political misinformation. We implement a two-step, chain-of-thought prompting approach, where models first identify credible sources for a given claim and then generate persuasive responses. Our findings suggest that models struggle to ground their responses in real news sources, and tend to prefer citing left-leaning sources. We also observe varying degrees of response diversity among models. Our findings highlight concerns about using LLMs for fact-checking through only prompt-engineering, emphasizing the need for more robust guardrails. Our results have implications for both researchers and non-technical users. |
| title | How LLMs Fail to Support Fact-Checking |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2503.01902 |