RAmBLA: A Framework for Evaluating the Reliability of LLMs as Assistants in the Biomedical Domain
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913276912205824 |
|---|---|
| author | Bolton, William James Poyiadzi, Rafael Morrell, Edward R. Bueno, Gabriela van Bergen Gonzalez Goetz, Lea |
| author_facet | Bolton, William James Poyiadzi, Rafael Morrell, Edward R. Bueno, Gabriela van Bergen Gonzalez Goetz, Lea |
| contents | Large Language Models (LLMs) increasingly support applications in a wide range of domains, some with potential high societal impact such as biomedicine, yet their reliability in realistic use cases is under-researched. In this work we introduce the Reliability AssesMent for Biomedical LLM Assistants (RAmBLA) framework and evaluate whether four state-of-the-art foundation LLMs can serve as reliable assistants in the biomedical domain. We identify prompt robustness, high recall, and a lack of hallucinations as necessary criteria for this use case. We design shortform tasks and tasks requiring LLM freeform responses mimicking real-world user interactions. We evaluate LLM performance using semantic similarity with a ground truth response, through an evaluator LLM. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_14578 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | RAmBLA: A Framework for Evaluating the Reliability of LLMs as Assistants in the Biomedical Domain Bolton, William James Poyiadzi, Rafael Morrell, Edward R. Bueno, Gabriela van Bergen Gonzalez Goetz, Lea Machine Learning Artificial Intelligence Large Language Models (LLMs) increasingly support applications in a wide range of domains, some with potential high societal impact such as biomedicine, yet their reliability in realistic use cases is under-researched. In this work we introduce the Reliability AssesMent for Biomedical LLM Assistants (RAmBLA) framework and evaluate whether four state-of-the-art foundation LLMs can serve as reliable assistants in the biomedical domain. We identify prompt robustness, high recall, and a lack of hallucinations as necessary criteria for this use case. We design shortform tasks and tasks requiring LLM freeform responses mimicking real-world user interactions. We evaluate LLM performance using semantic similarity with a ground truth response, through an evaluator LLM. |
| title | RAmBLA: A Framework for Evaluating the Reliability of LLMs as Assistants in the Biomedical Domain |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2403.14578 |