Faithfulness vs. Safety: Evaluating LLM Behavior Under Counterfactual Medical Evidence
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914487804624896 |
|---|---|
| author | Mo, Kaijie Venkatayogi, Siddhartha Shaib, Chantal Kouzy, Ramez Xu, Wei Wallace, Byron C. Li, Junyi Jessy |
| author_facet | Mo, Kaijie Venkatayogi, Siddhartha Shaib, Chantal Kouzy, Ramez Xu, Wei Wallace, Byron C. Li, Junyi Jessy |
| contents | In high-stakes domains like medicine, it may be generally desirable for models to faithfully adhere to the context provided. But what happens if the context does not align with model priors or safety protocols? In this paper, we investigate how LLMs behave and reason when presented with counterfactual (or even adversarial) medical evidence. We first construct MedCounterFact, a counterfactual medical QA dataset that requires the models to answer clinical comparison questions (i.e., judge the efficacy of certain treatments, with evidence consisting of randomized controlled trials provided as context). In MedCounterFact, real-world medical interventions within the questions and evidence are systematically replaced with four types of counterfactual stimuli, ranging from unknown words to toxic substances. Our evaluation across multiple frontier LLMs on MedCounterFact reveals that in the presence of counterfactual evidence, existing models overwhelmingly accept such "evidence" at face value even when it is dangerous or implausible, and provide confident and uncaveated answers. While it may be prudent to draw a boundary between faithfulness and safety, our findings suggest that models arguably overemphasize the former. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_11886 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Faithfulness vs. Safety: Evaluating LLM Behavior Under Counterfactual Medical Evidence Mo, Kaijie Venkatayogi, Siddhartha Shaib, Chantal Kouzy, Ramez Xu, Wei Wallace, Byron C. Li, Junyi Jessy Computation and Language In high-stakes domains like medicine, it may be generally desirable for models to faithfully adhere to the context provided. But what happens if the context does not align with model priors or safety protocols? In this paper, we investigate how LLMs behave and reason when presented with counterfactual (or even adversarial) medical evidence. We first construct MedCounterFact, a counterfactual medical QA dataset that requires the models to answer clinical comparison questions (i.e., judge the efficacy of certain treatments, with evidence consisting of randomized controlled trials provided as context). In MedCounterFact, real-world medical interventions within the questions and evidence are systematically replaced with four types of counterfactual stimuli, ranging from unknown words to toxic substances. Our evaluation across multiple frontier LLMs on MedCounterFact reveals that in the presence of counterfactual evidence, existing models overwhelmingly accept such "evidence" at face value even when it is dangerous or implausible, and provide confident and uncaveated answers. While it may be prudent to draw a boundary between faithfulness and safety, our findings suggest that models arguably overemphasize the former. |
| title | Faithfulness vs. Safety: Evaluating LLM Behavior Under Counterfactual Medical Evidence |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2601.11886 |