Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2509.24506 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908882604916736 |
|---|---|
| author | Hamna, Hamna Bhat, Gayatri Mukherjee, Sourabrata Lalani, Faisal Hadfield, Evan Siddarth, Divya Bali, Kalika Sitaram, Sunayana |
| author_facet | Hamna, Hamna Bhat, Gayatri Mukherjee, Sourabrata Lalani, Faisal Hadfield, Evan Siddarth, Divya Bali, Kalika Sitaram, Sunayana |
| contents | Large Language Models (LLMs) are typically evaluated through general or domain-specific benchmarks testing capabilities that often lack grounding in the lived realities of end users. Critical domains such as healthcare require evaluations that extend beyond artificial or simulated tasks to reflect the everyday needs, cultural practices, and nuanced contexts of communities. We propose Samiksha, a community-driven evaluation pipeline co-created with civil-society organizations (CSOs) and community members. Our approach enables scalable, automated benchmarking through a culturally aware, community-driven pipeline in which community feedback informs what to evaluate, how the benchmark is built, and how outputs are scored. We demonstrate this approach in the health domain in India. Our analysis highlights how current multilingual LLMs address nuanced community health queries, while also offering a scalable pathway for contextually grounded and inclusive LLM evaluation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_24506 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Building Benchmarks from the Ground Up: Community-Centered Evaluation of LLMs in Healthcare Chatbot Settings Hamna, Hamna Bhat, Gayatri Mukherjee, Sourabrata Lalani, Faisal Hadfield, Evan Siddarth, Divya Bali, Kalika Sitaram, Sunayana Computation and Language Large Language Models (LLMs) are typically evaluated through general or domain-specific benchmarks testing capabilities that often lack grounding in the lived realities of end users. Critical domains such as healthcare require evaluations that extend beyond artificial or simulated tasks to reflect the everyday needs, cultural practices, and nuanced contexts of communities. We propose Samiksha, a community-driven evaluation pipeline co-created with civil-society organizations (CSOs) and community members. Our approach enables scalable, automated benchmarking through a culturally aware, community-driven pipeline in which community feedback informs what to evaluate, how the benchmark is built, and how outputs are scored. We demonstrate this approach in the health domain in India. Our analysis highlights how current multilingual LLMs address nuanced community health queries, while also offering a scalable pathway for contextually grounded and inclusive LLM evaluation. |
| title | Building Benchmarks from the Ground Up: Community-Centered Evaluation of LLMs in Healthcare Chatbot Settings |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2509.24506 |