A Mixed-Methods Evaluation of LLM-Based Chatbots for Menopause
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917914599227392 |
|---|---|
| author | Deva, Roshini S, Manvi Zhou, Jasmine Chahine, Elizabeth Britton Davenport-Nicholson, Agena Kaonga, Nadi Nina Bozkurt, Selen Ismail, Azra |
| author_facet | Deva, Roshini S, Manvi Zhou, Jasmine Chahine, Elizabeth Britton Davenport-Nicholson, Agena Kaonga, Nadi Nina Bozkurt, Selen Ismail, Azra |
| contents | The integration of Large Language Models (LLMs) into healthcare settings has gained significant attention, particularly for question-answering tasks. Given the high-stakes nature of healthcare, it is essential to ensure that LLM-generated content is accurate and reliable to prevent adverse outcomes. However, the development of robust evaluation metrics and methodologies remains a matter of much debate. We examine the performance of publicly available LLM-based chatbots for menopause-related queries, using a mixed-methods approach to evaluate safety, consensus, objectivity, reproducibility, and explainability. Our findings highlight the promise and limitations of traditional evaluation metrics for sensitive health topics. We propose the need for customized and ethically grounded evaluation frameworks to assess LLMs to advance safe and effective use in healthcare. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_03579 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | A Mixed-Methods Evaluation of LLM-Based Chatbots for Menopause Deva, Roshini S, Manvi Zhou, Jasmine Chahine, Elizabeth Britton Davenport-Nicholson, Agena Kaonga, Nadi Nina Bozkurt, Selen Ismail, Azra Computers and Society Human-Computer Interaction The integration of Large Language Models (LLMs) into healthcare settings has gained significant attention, particularly for question-answering tasks. Given the high-stakes nature of healthcare, it is essential to ensure that LLM-generated content is accurate and reliable to prevent adverse outcomes. However, the development of robust evaluation metrics and methodologies remains a matter of much debate. We examine the performance of publicly available LLM-based chatbots for menopause-related queries, using a mixed-methods approach to evaluate safety, consensus, objectivity, reproducibility, and explainability. Our findings highlight the promise and limitations of traditional evaluation metrics for sensitive health topics. We propose the need for customized and ethically grounded evaluation frameworks to assess LLMs to advance safe and effective use in healthcare. |
| title | A Mixed-Methods Evaluation of LLM-Based Chatbots for Menopause |
| topic | Computers and Society Human-Computer Interaction |
| url | https://arxiv.org/abs/2502.03579 |