Healthy LLMs? Benchmarking LLM Knowledge of UK Government Public Health Information
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915844105175040 |
|---|---|
| author | Harris, Joshua Grayson, Fan Feldman, Felix Laurence, Timothy Nonnenmacher, Toby Higgins, Oliver Loman, Leo Patel, Selina Finnie, Thomas Collins, Samuel Borowitz, Michael |
| author_facet | Harris, Joshua Grayson, Fan Feldman, Felix Laurence, Timothy Nonnenmacher, Toby Higgins, Oliver Loman, Leo Patel, Selina Finnie, Thomas Collins, Samuel Borowitz, Michael |
| contents | As Large Language Models (LLMs) become widely accessible, a detailed understanding of their knowledge within specific domains becomes necessary for successful real world use. This is particularly critical in the domains of medicine and public health, where failure to retrieve relevant, accurate, and current information could significantly impact UK residents. However, while there are a number of LLM benchmarks in the medical domain, currently little is known about LLM knowledge within the field of public health. To address this issue, this paper introduces a new benchmark, PubHealthBench, with over 8000 questions for evaluating LLMs' Multiple Choice Question Answering (MCQA) and free form responses to public health queries. To create PubHealthBench we extract free text from 687 current UK government guidance documents and implement an automated pipeline for generating MCQA samples. Assessing 24 LLMs on PubHealthBench we find the latest proprietary LLMs (GPT-4.5, GPT-4.1 and o1) have a high degree of knowledge, achieving >90% accuracy in the MCQA setup, and outperform humans with cursory search engine use. However, in the free form setup we see lower performance with no model scoring >75%. Therefore, while there are promising signs that state of the art (SOTA) LLMs are an increasingly accurate source of public health information, additional safeguards or tools may still be needed when providing free form responses. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_06046 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Healthy LLMs? Benchmarking LLM Knowledge of UK Government Public Health Information Harris, Joshua Grayson, Fan Feldman, Felix Laurence, Timothy Nonnenmacher, Toby Higgins, Oliver Loman, Leo Patel, Selina Finnie, Thomas Collins, Samuel Borowitz, Michael Computation and Language Machine Learning 68T50 As Large Language Models (LLMs) become widely accessible, a detailed understanding of their knowledge within specific domains becomes necessary for successful real world use. This is particularly critical in the domains of medicine and public health, where failure to retrieve relevant, accurate, and current information could significantly impact UK residents. However, while there are a number of LLM benchmarks in the medical domain, currently little is known about LLM knowledge within the field of public health. To address this issue, this paper introduces a new benchmark, PubHealthBench, with over 8000 questions for evaluating LLMs' Multiple Choice Question Answering (MCQA) and free form responses to public health queries. To create PubHealthBench we extract free text from 687 current UK government guidance documents and implement an automated pipeline for generating MCQA samples. Assessing 24 LLMs on PubHealthBench we find the latest proprietary LLMs (GPT-4.5, GPT-4.1 and o1) have a high degree of knowledge, achieving >90% accuracy in the MCQA setup, and outperform humans with cursory search engine use. However, in the free form setup we see lower performance with no model scoring >75%. Therefore, while there are promising signs that state of the art (SOTA) LLMs are an increasingly accurate source of public health information, additional safeguards or tools may still be needed when providing free form responses. |
| title | Healthy LLMs? Benchmarking LLM Knowledge of UK Government Public Health Information |
| topic | Computation and Language Machine Learning 68T50 |
| url | https://arxiv.org/abs/2505.06046 |