Guardado en:
| Autores principales: | , , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | https://arxiv.org/abs/2512.17028 |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866908721617043456 |
|---|---|
| author | Gruber, Victoria-Elisabeth Marinescu, Razvan Fajardo, Diego Nassar, Amin H. Arkfeld, Christopher Ludlow, Alexandria Patel, Shama Samaei, Mehrnoosh Klug, Valerie Huber, Anna Gühner, Marcel Orfila, Albert Botta i Lagoja, Irene Tarr, Kimya Larson, Haleigh Howard, Mary Beth |
| author_facet | Gruber, Victoria-Elisabeth Marinescu, Razvan Fajardo, Diego Nassar, Amin H. Arkfeld, Christopher Ludlow, Alexandria Patel, Shama Samaei, Mehrnoosh Klug, Valerie Huber, Anna Gühner, Marcel Orfila, Albert Botta i Lagoja, Irene Tarr, Kimya Larson, Haleigh Howard, Mary Beth |
| contents | As large language models (LLMs) become primary sources of health information for millions, their accuracy in women's health remains critically unexamined. We introduce the Women's Health Benchmark (WHB), the first benchmark evaluating LLM performance specifically in women's health. Our benchmark comprises 96 rigorously validated model stumps covering five medical specialties (obstetrics and gynecology, emergency medicine, primary care, oncology, and neurology), three query types (patient query, clinician query, and evidence/policy query), and eight error types (dosage/medication errors, missing critical information, outdated guidelines/treatment recommendations, incorrect treatment advice, incorrect factual information, missing/incorrect differential diagnosis, missed urgency, and inappropriate recommendations). We evaluated 13 state-of-the-art LLMs and revealed alarming gaps: current models show approximately 60\% failure rates on the women's health benchmark, with performance varying dramatically across specialties and error types. Notably, models universally struggle with "missed urgency" indicators, while newer models like GPT-5 show significant improvements in avoiding inappropriate recommendations. Our findings underscore that AI chatbots are not yet fully able of providing reliable advice in women's health. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_17028 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | A Women's Health Benchmark for Large Language Models Gruber, Victoria-Elisabeth Marinescu, Razvan Fajardo, Diego Nassar, Amin H. Arkfeld, Christopher Ludlow, Alexandria Patel, Shama Samaei, Mehrnoosh Klug, Valerie Huber, Anna Gühner, Marcel Orfila, Albert Botta i Lagoja, Irene Tarr, Kimya Larson, Haleigh Howard, Mary Beth Computation and Language Artificial Intelligence Machine Learning As large language models (LLMs) become primary sources of health information for millions, their accuracy in women's health remains critically unexamined. We introduce the Women's Health Benchmark (WHB), the first benchmark evaluating LLM performance specifically in women's health. Our benchmark comprises 96 rigorously validated model stumps covering five medical specialties (obstetrics and gynecology, emergency medicine, primary care, oncology, and neurology), three query types (patient query, clinician query, and evidence/policy query), and eight error types (dosage/medication errors, missing critical information, outdated guidelines/treatment recommendations, incorrect treatment advice, incorrect factual information, missing/incorrect differential diagnosis, missed urgency, and inappropriate recommendations). We evaluated 13 state-of-the-art LLMs and revealed alarming gaps: current models show approximately 60\% failure rates on the women's health benchmark, with performance varying dramatically across specialties and error types. Notably, models universally struggle with "missed urgency" indicators, while newer models like GPT-5 show significant improvements in avoiding inappropriate recommendations. Our findings underscore that AI chatbots are not yet fully able of providing reliable advice in women's health. |
| title | A Women's Health Benchmark for Large Language Models |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2512.17028 |