Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2411.15640 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918145186332672 |
|---|---|
| author | Olatunji, Tobi Nimo, Charles Owodunni, Abraham Abdullahi, Tassallah Ayodele, Emmanuel Sanni, Mardhiyah Aka, Chinemelu Omofoye, Folafunmi Yuehgoh, Foutse Faniran, Timothy Dossou, Bonaventure F. P. Yekini, Moshood Kemp, Jonas Heller, Katherine Omeke, Jude Chidubem MD, Chidi Asuzu Etori, Naome A. Ndiaye, Aimérou Okoh, Ifeoma Ocansey, Evans Doe Kinara, Wendy Best, Michael Essa, Irfan Moore, Stephen Edward Fourie, Chris Asiedu, Mercy Nyamewaa |
| author_facet | Olatunji, Tobi Nimo, Charles Owodunni, Abraham Abdullahi, Tassallah Ayodele, Emmanuel Sanni, Mardhiyah Aka, Chinemelu Omofoye, Folafunmi Yuehgoh, Foutse Faniran, Timothy Dossou, Bonaventure F. P. Yekini, Moshood Kemp, Jonas Heller, Katherine Omeke, Jude Chidubem MD, Chidi Asuzu Etori, Naome A. Ndiaye, Aimérou Okoh, Ifeoma Ocansey, Evans Doe Kinara, Wendy Best, Michael Essa, Irfan Moore, Stephen Edward Fourie, Chris Asiedu, Mercy Nyamewaa |
| contents | Recent advancements in large language model(LLM) performance on medical multiple choice question (MCQ) benchmarks have stimulated interest from healthcare providers and patients globally. Particularly in low-and middle-income countries (LMICs) facing acute physician shortages and lack of specialists, LLMs offer a potentially scalable pathway to enhance healthcare access and reduce costs. However, their effectiveness in the Global South, especially across the African continent, remains to be established. In this work, we introduce AfriMed-QA, the first large scale Pan-African English multi-specialty medical Question-Answering (QA) dataset, 15,000 questions (open and closed-ended) sourced from over 60 medical schools across 16 countries, covering 32 medical specialties. We further evaluate 30 LLMs across multiple axes including correctness and demographic bias. Our findings show significant performance variation across specialties and geographies, MCQ performance clearly lags USMLE (MedQA). We find that biomedical LLMs underperform general models and smaller edge-friendly LLMs struggle to achieve a passing score. Interestingly, human evaluations show a consistent consumer preference for LLM answers and explanations when compared with clinician answers. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_15640 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | AfriMed-QA: A Pan-African, Multi-Specialty, Medical Question-Answering Benchmark Dataset Olatunji, Tobi Nimo, Charles Owodunni, Abraham Abdullahi, Tassallah Ayodele, Emmanuel Sanni, Mardhiyah Aka, Chinemelu Omofoye, Folafunmi Yuehgoh, Foutse Faniran, Timothy Dossou, Bonaventure F. P. Yekini, Moshood Kemp, Jonas Heller, Katherine Omeke, Jude Chidubem MD, Chidi Asuzu Etori, Naome A. Ndiaye, Aimérou Okoh, Ifeoma Ocansey, Evans Doe Kinara, Wendy Best, Michael Essa, Irfan Moore, Stephen Edward Fourie, Chris Asiedu, Mercy Nyamewaa Computation and Language Recent advancements in large language model(LLM) performance on medical multiple choice question (MCQ) benchmarks have stimulated interest from healthcare providers and patients globally. Particularly in low-and middle-income countries (LMICs) facing acute physician shortages and lack of specialists, LLMs offer a potentially scalable pathway to enhance healthcare access and reduce costs. However, their effectiveness in the Global South, especially across the African continent, remains to be established. In this work, we introduce AfriMed-QA, the first large scale Pan-African English multi-specialty medical Question-Answering (QA) dataset, 15,000 questions (open and closed-ended) sourced from over 60 medical schools across 16 countries, covering 32 medical specialties. We further evaluate 30 LLMs across multiple axes including correctness and demographic bias. Our findings show significant performance variation across specialties and geographies, MCQ performance clearly lags USMLE (MedQA). We find that biomedical LLMs underperform general models and smaller edge-friendly LLMs struggle to achieve a passing score. Interestingly, human evaluations show a consistent consumer preference for LLM answers and explanations when compared with clinician answers. |
| title | AfriMed-QA: A Pan-African, Multi-Specialty, Medical Question-Answering Benchmark Dataset |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2411.15640 |