Truth, Trust, and Trouble: Medical AI on the Edge
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866918532896260096 |
|---|---|
| author | Azeez, Mohammad Anas Ali, Rafiq Shabbir, Ebad Siddiqui, Zohaib Hasan Kashyap, Gautam Siddharth Gao, Jiechao Naseem, Usman |
| author_facet | Azeez, Mohammad Anas Ali, Rafiq Shabbir, Ebad Siddiqui, Zohaib Hasan Kashyap, Gautam Siddharth Gao, Jiechao Naseem, Usman |
| contents | Large Language Models (LLMs) hold significant promise for transforming digital health by enabling automated medical question answering. However, ensuring these models meet critical industry standards for factual accuracy, usefulness, and safety remains a challenge, especially for open-source solutions. We present a rigorous benchmarking framework using a dataset of over 1,000 health questions. We assess model performance across honesty, helpfulness, and harmlessness. Our results highlight trade-offs between factual reliability and safety among evaluated models -- Mistral-7B, BioMistral-7B-DARE, and AlpaCare-13B. AlpaCare-13B achieves the highest accuracy (91.7%) and harmlessness (0.92), while domain-specific tuning in BioMistral-7B-DARE boosts safety (0.90) despite its smaller scale. Few-shot prompting improves accuracy from 78% to 85%, and all models show reduced helpfulness on complex queries, highlighting ongoing challenges in clinical QA. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_02983 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Truth, Trust, and Trouble: Medical AI on the Edge Azeez, Mohammad Anas Ali, Rafiq Shabbir, Ebad Siddiqui, Zohaib Hasan Kashyap, Gautam Siddharth Gao, Jiechao Naseem, Usman Computation and Language Artificial Intelligence Large Language Models (LLMs) hold significant promise for transforming digital health by enabling automated medical question answering. However, ensuring these models meet critical industry standards for factual accuracy, usefulness, and safety remains a challenge, especially for open-source solutions. We present a rigorous benchmarking framework using a dataset of over 1,000 health questions. We assess model performance across honesty, helpfulness, and harmlessness. Our results highlight trade-offs between factual reliability and safety among evaluated models -- Mistral-7B, BioMistral-7B-DARE, and AlpaCare-13B. AlpaCare-13B achieves the highest accuracy (91.7%) and harmlessness (0.92), while domain-specific tuning in BioMistral-7B-DARE boosts safety (0.90) despite its smaller scale. Few-shot prompting improves accuracy from 78% to 85%, and all models show reduced helpfulness on complex queries, highlighting ongoing challenges in clinical QA. |
| title | Truth, Trust, and Trouble: Medical AI on the Edge |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2507.02983 |