Faithful or Just Plausible? Evaluating the Faithfulness of Closed-Source LLMs in Medical Reasoning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911672643354624 |
|---|---|
| author | Afolabi, Halimat Afolabi, Zainab Friel, Elizabeth Roberts, Jude Ji-Xu, Antonio Chen, Lloyd Ogbomo, Egheosa Imevbore, Emiliomo Eneje, Phil Ouahidi, Wissal El Sohal, Aaron Kennan, Alisa Srivastava, Shreya Vairavan, Anirudh Napitu, Laura McClure, Katie |
| author_facet | Afolabi, Halimat Afolabi, Zainab Friel, Elizabeth Roberts, Jude Ji-Xu, Antonio Chen, Lloyd Ogbomo, Egheosa Imevbore, Emiliomo Eneje, Phil Ouahidi, Wissal El Sohal, Aaron Kennan, Alisa Srivastava, Shreya Vairavan, Anirudh Napitu, Laura McClure, Katie |
| contents | Closed-source large language models (LLMs), such as ChatGPT and Gemini, are increasingly consulted for medical advice, yet their explanations may appear plausible while failing to reflect the model's underlying reasoning process. This gap poses serious risks as patients and clinicians may trust coherent but misleading explanations. We conduct a systematic black-box evaluation of faithfulness in medical reasoning among three widely used closed-source LLMs. Our study consists of three perturbation-based probes: (1) causal ablation, testing whether stated chain-of-thought (CoT) reasoning causally influences predictions; (2) positional bias, examining whether models create post-hoc justifications for answers driven by input positioning; and (3) hint injection, testing susceptibility to external suggestions. We complement these quantitative probes with a small-scale human evaluation of model responses to patient-style medical queries to examine concordance between physician assessments of explanation faithfulness and layperson perceptions of trustworthiness. We find that CoT reasoning steps often do not causally drive predictions, and models readily incorporate external hints without acknowledgment. In contrast, positional biases showed minimal impact in this setting. These results underscore that faithfulness, not just accuracy, must be central in evaluating LLMs for medicine, to ensure both public protection and safe clinical deployment. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_13988 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Faithful or Just Plausible? Evaluating the Faithfulness of Closed-Source LLMs in Medical Reasoning Afolabi, Halimat Afolabi, Zainab Friel, Elizabeth Roberts, Jude Ji-Xu, Antonio Chen, Lloyd Ogbomo, Egheosa Imevbore, Emiliomo Eneje, Phil Ouahidi, Wissal El Sohal, Aaron Kennan, Alisa Srivastava, Shreya Vairavan, Anirudh Napitu, Laura McClure, Katie Artificial Intelligence Machine Learning Closed-source large language models (LLMs), such as ChatGPT and Gemini, are increasingly consulted for medical advice, yet their explanations may appear plausible while failing to reflect the model's underlying reasoning process. This gap poses serious risks as patients and clinicians may trust coherent but misleading explanations. We conduct a systematic black-box evaluation of faithfulness in medical reasoning among three widely used closed-source LLMs. Our study consists of three perturbation-based probes: (1) causal ablation, testing whether stated chain-of-thought (CoT) reasoning causally influences predictions; (2) positional bias, examining whether models create post-hoc justifications for answers driven by input positioning; and (3) hint injection, testing susceptibility to external suggestions. We complement these quantitative probes with a small-scale human evaluation of model responses to patient-style medical queries to examine concordance between physician assessments of explanation faithfulness and layperson perceptions of trustworthiness. We find that CoT reasoning steps often do not causally drive predictions, and models readily incorporate external hints without acknowledgment. In contrast, positional biases showed minimal impact in this setting. These results underscore that faithfulness, not just accuracy, must be central in evaluating LLMs for medicine, to ensure both public protection and safe clinical deployment. |
| title | Faithful or Just Plausible? Evaluating the Faithfulness of Closed-Source LLMs in Medical Reasoning |
| topic | Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2603.13988 |