Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2409.09201 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929677069713408 |
|---|---|
| author | Asiedu, Mercy Tomasev, Nenad Ghate, Chintan Tiyasirichokchai, Tiya Dieng, Awa Akande, Oluwatosin Siwo, Geoffrey Adudans, Steve Aitkins, Sylvanus Ehiakhamen, Odianosen Ndombi, Eric Heller, Katherine |
| author_facet | Asiedu, Mercy Tomasev, Nenad Ghate, Chintan Tiyasirichokchai, Tiya Dieng, Awa Akande, Oluwatosin Siwo, Geoffrey Adudans, Steve Aitkins, Sylvanus Ehiakhamen, Odianosen Ndombi, Eric Heller, Katherine |
| contents | While large language models (LLMs) have shown promise for medical question answering, there is limited work focused on tropical and infectious disease-specific exploration. We build on an opensource tropical and infectious diseases (TRINDs) dataset, expanding it to include demographic and semantic clinical and consumer augmentations yielding 11000+ prompts. We evaluate LLM performance on these, comparing generalist and medical LLMs, as well as LLM outcomes to human experts. We demonstrate through systematic experimentation, the benefit of contextual information such as demographics, location, gender, risk factors for optimal LLM response. Finally we develop a prototype of TRINDs-LM, a research tool that provides a playground to navigate how context impacts LLM outputs for health. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_09201 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Contextual Evaluation of Large Language Models for Classifying Tropical and Infectious Diseases Asiedu, Mercy Tomasev, Nenad Ghate, Chintan Tiyasirichokchai, Tiya Dieng, Awa Akande, Oluwatosin Siwo, Geoffrey Adudans, Steve Aitkins, Sylvanus Ehiakhamen, Odianosen Ndombi, Eric Heller, Katherine Computation and Language Artificial Intelligence While large language models (LLMs) have shown promise for medical question answering, there is limited work focused on tropical and infectious disease-specific exploration. We build on an opensource tropical and infectious diseases (TRINDs) dataset, expanding it to include demographic and semantic clinical and consumer augmentations yielding 11000+ prompts. We evaluate LLM performance on these, comparing generalist and medical LLMs, as well as LLM outcomes to human experts. We demonstrate through systematic experimentation, the benefit of contextual information such as demographics, location, gender, risk factors for optimal LLM response. Finally we develop a prototype of TRINDs-LM, a research tool that provides a playground to navigate how context impacts LLM outputs for health. |
| title | Contextual Evaluation of Large Language Models for Classifying Tropical and Infectious Diseases |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2409.09201 |