Evaluating Large Language Models' Responses to Sexual and Reproductive Health Queries in Nepali
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910066330828800 |
|---|---|
| author | Sharma, Medha Khadka, Supriya Aryal, Udit Chandra Bhatta, Bishnu Hari Bhattarai, Bijayan Dahal, Santosh Gautam, Kamal Joshi, Pushpa Kafle, Saugat Khadka, Shristi Khadka, Shushila Lamichhane, Binod Lamichhane, Shilpa Parajuli, Anusha Pokharel, Sabina Sitaula, Suvekshya Verma, Neha Khanal, Bishesh |
| author_facet | Sharma, Medha Khadka, Supriya Aryal, Udit Chandra Bhatta, Bishnu Hari Bhattarai, Bijayan Dahal, Santosh Gautam, Kamal Joshi, Pushpa Kafle, Saugat Khadka, Shristi Khadka, Shushila Lamichhane, Binod Lamichhane, Shilpa Parajuli, Anusha Pokharel, Sabina Sitaula, Suvekshya Verma, Neha Khanal, Bishesh |
| contents | As Large Language Models (LLMs) become integrated into daily life, they are increasingly used for personal queries, including Sexual and Reproductive Health (SRH), allowing users to chat anonymously without fear of judgment. However, current evaluation methods primarily focus on accuracy, often for objective queries in high-resource languages, and lack criteria to assess usability and safety, especially for low-resource languages and culturally sensitive domains like SRH. This paper introduces LLM Evaluation Framework (LEAF), that conducts assessments across multiple criteria: accuracy, language, usability gaps (including relevance, adequacy, and cultural appropriateness), and safety gaps (safety, sensitivity, and confidentiality). Using the LEAF framework, we assessed 14K SRH queries in Nepali from over 9K users. Responses were manually annotated by SRH experts according to the framework. Results revealed that only 35.1% of the responses were "proper", meaning they were accurate, adequate and had no major usability or safety related gaps. Insights include differences in performance between ChatGPT versions, such as similar accuracy but varying usability and safety aspects. This evaluation highlights significant limitations of current LLMs and underscores the need for improvement. The LEAF Framework is adaptable across domains and languages, particularly where usability and safety are critical, offering a pathway to better address sensitive topics. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_22291 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Evaluating Large Language Models' Responses to Sexual and Reproductive Health Queries in Nepali Sharma, Medha Khadka, Supriya Aryal, Udit Chandra Bhatta, Bishnu Hari Bhattarai, Bijayan Dahal, Santosh Gautam, Kamal Joshi, Pushpa Kafle, Saugat Khadka, Shristi Khadka, Shushila Lamichhane, Binod Lamichhane, Shilpa Parajuli, Anusha Pokharel, Sabina Sitaula, Suvekshya Verma, Neha Khanal, Bishesh Computation and Language As Large Language Models (LLMs) become integrated into daily life, they are increasingly used for personal queries, including Sexual and Reproductive Health (SRH), allowing users to chat anonymously without fear of judgment. However, current evaluation methods primarily focus on accuracy, often for objective queries in high-resource languages, and lack criteria to assess usability and safety, especially for low-resource languages and culturally sensitive domains like SRH. This paper introduces LLM Evaluation Framework (LEAF), that conducts assessments across multiple criteria: accuracy, language, usability gaps (including relevance, adequacy, and cultural appropriateness), and safety gaps (safety, sensitivity, and confidentiality). Using the LEAF framework, we assessed 14K SRH queries in Nepali from over 9K users. Responses were manually annotated by SRH experts according to the framework. Results revealed that only 35.1% of the responses were "proper", meaning they were accurate, adequate and had no major usability or safety related gaps. Insights include differences in performance between ChatGPT versions, such as similar accuracy but varying usability and safety aspects. This evaluation highlights significant limitations of current LLMs and underscores the need for improvement. The LEAF Framework is adaptable across domains and languages, particularly where usability and safety are critical, offering a pathway to better address sensitive topics. |
| title | Evaluating Large Language Models' Responses to Sexual and Reproductive Health Queries in Nepali |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2603.22291 |