DiversityMedQA: Assessing Demographic Biases in Medical Diagnosis using Large Language Models
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916510549671936 |
|---|---|
| author | Rawat, Rajat McBride, Hudson Nirmal, Dhiyaan Ghosh, Rajarshi Moon, Jong Alamuri, Dhruv O'Brien, Sean Zhu, Kevin |
| author_facet | Rawat, Rajat McBride, Hudson Nirmal, Dhiyaan Ghosh, Rajarshi Moon, Jong Alamuri, Dhruv O'Brien, Sean Zhu, Kevin |
| contents | As large language models (LLMs) gain traction in healthcare, concerns about their susceptibility to demographic biases are growing. We introduce {DiversityMedQA}, a novel benchmark designed to assess LLM responses to medical queries across diverse patient demographics, such as gender and ethnicity. By perturbing questions from the MedQA dataset, which comprises medical board exam questions, we created a benchmark that captures the nuanced differences in medical diagnosis across varying patient profiles. Our findings reveal notable discrepancies in model performance when tested against these demographic variations. Furthermore, to ensure the perturbations were accurate, we also propose a filtering strategy that validates each perturbation. By releasing DiversityMedQA, we provide a resource for evaluating and mitigating demographic bias in LLM medical diagnoses. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_01497 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | DiversityMedQA: Assessing Demographic Biases in Medical Diagnosis using Large Language Models Rawat, Rajat McBride, Hudson Nirmal, Dhiyaan Ghosh, Rajarshi Moon, Jong Alamuri, Dhruv O'Brien, Sean Zhu, Kevin Computation and Language As large language models (LLMs) gain traction in healthcare, concerns about their susceptibility to demographic biases are growing. We introduce {DiversityMedQA}, a novel benchmark designed to assess LLM responses to medical queries across diverse patient demographics, such as gender and ethnicity. By perturbing questions from the MedQA dataset, which comprises medical board exam questions, we created a benchmark that captures the nuanced differences in medical diagnosis across varying patient profiles. Our findings reveal notable discrepancies in model performance when tested against these demographic variations. Furthermore, to ensure the perturbations were accurate, we also propose a filtering strategy that validates each perturbation. By releasing DiversityMedQA, we provide a resource for evaluating and mitigating demographic bias in LLM medical diagnoses. |
| title | DiversityMedQA: Assessing Demographic Biases in Medical Diagnosis using Large Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2409.01497 |