Automatic Evaluation of Healthcare LLMs Beyond Question-Answering
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916622375059456 |
|---|---|
| author | Arias-Duart, Anna Martin-Torres, Pablo Agustin Hinjos, Daniel Bernabeu-Perez, Pablo Ganzabal, Lucia Urcelay Mallo, Marta Gonzalez Gururajan, Ashwin Kumar Lopez-Cuena, Enrique Alvarez-Napagao, Sergio Garcia-Gasulla, Dario |
| author_facet | Arias-Duart, Anna Martin-Torres, Pablo Agustin Hinjos, Daniel Bernabeu-Perez, Pablo Ganzabal, Lucia Urcelay Mallo, Marta Gonzalez Gururajan, Ashwin Kumar Lopez-Cuena, Enrique Alvarez-Napagao, Sergio Garcia-Gasulla, Dario |
| contents | Current Large Language Models (LLMs) benchmarks are often based on open-ended or close-ended QA evaluations, avoiding the requirement of human labor. Close-ended measurements evaluate the factuality of responses but lack expressiveness. Open-ended capture the model's capacity to produce discourse responses but are harder to assess for correctness. These two approaches are commonly used, either independently or together, though their relationship remains poorly understood. This work is focused on the healthcare domain, where both factuality and discourse matter greatly. It introduces a comprehensive, multi-axis suite for healthcare LLM evaluation, exploring correlations between open and close benchmarks and metrics. Findings include blind spots and overlaps in current methodologies. As an updated sanity check, we release a new medical benchmark --CareQA-- with both open and closed variants. Finally, we propose a novel metric for open-ended evaluations -- Relaxed Perplexity -- to mitigate the identified limitations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_06666 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Automatic Evaluation of Healthcare LLMs Beyond Question-Answering Arias-Duart, Anna Martin-Torres, Pablo Agustin Hinjos, Daniel Bernabeu-Perez, Pablo Ganzabal, Lucia Urcelay Mallo, Marta Gonzalez Gururajan, Ashwin Kumar Lopez-Cuena, Enrique Alvarez-Napagao, Sergio Garcia-Gasulla, Dario Computation and Language Artificial Intelligence Current Large Language Models (LLMs) benchmarks are often based on open-ended or close-ended QA evaluations, avoiding the requirement of human labor. Close-ended measurements evaluate the factuality of responses but lack expressiveness. Open-ended capture the model's capacity to produce discourse responses but are harder to assess for correctness. These two approaches are commonly used, either independently or together, though their relationship remains poorly understood. This work is focused on the healthcare domain, where both factuality and discourse matter greatly. It introduces a comprehensive, multi-axis suite for healthcare LLM evaluation, exploring correlations between open and close benchmarks and metrics. Findings include blind spots and overlaps in current methodologies. As an updated sanity check, we release a new medical benchmark --CareQA-- with both open and closed variants. Finally, we propose a novel metric for open-ended evaluations -- Relaxed Perplexity -- to mitigate the identified limitations. |
| title | Automatic Evaluation of Healthcare LLMs Beyond Question-Answering |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2502.06666 |