Evaluating Large Language Models for Public Health Classification and Extraction Tasks
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910835174014976 |
|---|---|
| author | Harris, Joshua Laurence, Timothy Loman, Leo Grayson, Fan Nonnenmacher, Toby Long, Harry WalsGriffith, Loes Douglas, Amy Fountain, Holly Georgiou, Stelios Hardstaff, Jo Hopkins, Kathryn Chi, Y-Ling Kuyumdzhieva, Galena Larkin, Lesley Collins, Samuel Mohammed, Hamish Finnie, Thomas Hounsome, Luke Borowitz, Michael Riley, Steven |
| author_facet | Harris, Joshua Laurence, Timothy Loman, Leo Grayson, Fan Nonnenmacher, Toby Long, Harry WalsGriffith, Loes Douglas, Amy Fountain, Holly Georgiou, Stelios Hardstaff, Jo Hopkins, Kathryn Chi, Y-Ling Kuyumdzhieva, Galena Larkin, Lesley Collins, Samuel Mohammed, Hamish Finnie, Thomas Hounsome, Luke Borowitz, Michael Riley, Steven |
| contents | Advances in Large Language Models (LLMs) have led to significant interest in their potential to support human experts across a range of domains, including public health. In this work we present automated evaluations of LLMs for public health tasks involving the classification and extraction of free text. We combine six externally annotated datasets with seven new internally annotated datasets to evaluate LLMs for processing text related to: health burden, epidemiological risk factors, and public health interventions. We evaluate eleven open-weight LLMs (7-123 billion parameters) across all tasks using zero-shot in-context learning. We find that Llama-3.3-70B-Instruct is the highest performing model, achieving the best results on 8/16 tasks (using micro-F1 scores). We see significant variation across tasks with all open-weight LLMs scoring below 60% micro-F1 on some challenging tasks, such as Contact Classification, while all LLMs achieve greater than 80% micro-F1 on others, such as GI Illness Classification. For a subset of 11 tasks, we also evaluate three GPT-4 and GPT-4o series models and find comparable results to Llama-3.3-70B-Instruct. Overall, based on these initial results we find promising signs that LLMs may be useful tools for public health experts to extract information from a wide variety of free text sources, and support public health surveillance, research, and interventions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_14766 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Evaluating Large Language Models for Public Health Classification and Extraction Tasks Harris, Joshua Laurence, Timothy Loman, Leo Grayson, Fan Nonnenmacher, Toby Long, Harry WalsGriffith, Loes Douglas, Amy Fountain, Holly Georgiou, Stelios Hardstaff, Jo Hopkins, Kathryn Chi, Y-Ling Kuyumdzhieva, Galena Larkin, Lesley Collins, Samuel Mohammed, Hamish Finnie, Thomas Hounsome, Luke Borowitz, Michael Riley, Steven Computation and Language Machine Learning 68T50 Advances in Large Language Models (LLMs) have led to significant interest in their potential to support human experts across a range of domains, including public health. In this work we present automated evaluations of LLMs for public health tasks involving the classification and extraction of free text. We combine six externally annotated datasets with seven new internally annotated datasets to evaluate LLMs for processing text related to: health burden, epidemiological risk factors, and public health interventions. We evaluate eleven open-weight LLMs (7-123 billion parameters) across all tasks using zero-shot in-context learning. We find that Llama-3.3-70B-Instruct is the highest performing model, achieving the best results on 8/16 tasks (using micro-F1 scores). We see significant variation across tasks with all open-weight LLMs scoring below 60% micro-F1 on some challenging tasks, such as Contact Classification, while all LLMs achieve greater than 80% micro-F1 on others, such as GI Illness Classification. For a subset of 11 tasks, we also evaluate three GPT-4 and GPT-4o series models and find comparable results to Llama-3.3-70B-Instruct. Overall, based on these initial results we find promising signs that LLMs may be useful tools for public health experts to extract information from a wide variety of free text sources, and support public health surveillance, research, and interventions. |
| title | Evaluating Large Language Models for Public Health Classification and Extraction Tasks |
| topic | Computation and Language Machine Learning 68T50 |
| url | https://arxiv.org/abs/2405.14766 |