Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2603.13168 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910051956948992 |
|---|---|
| author | Jha, Smriti Jain, Vidhi Xu, Jianyu Liu, Grace Ramesh, Sowmya Nagpal, Jitender Chapman, Gretchen Bellows, Benjamin Goyal, Siddhartha Singh, Aarti Wilder, Bryan |
| author_facet | Jha, Smriti Jain, Vidhi Xu, Jianyu Liu, Grace Ramesh, Sowmya Nagpal, Jitender Chapman, Gretchen Bellows, Benjamin Goyal, Siddhartha Singh, Aarti Wilder, Bryan |
| contents | The ability to provide trustworthy maternal health information using phone-based chatbots can have a significant impact, particularly in low-resource settings where users have low health literacy and limited access to care. However, deploying such systems is technically challenging: user queries are short, underspecified, and code-mixed across languages, answers require regional context-specific grounding, and partial or missing symptom context makes safe routing decisions difficult.
We present a chatbot for maternal health in India developed through a partnership between academic researchers, a health tech company, a public health nonprofit, and a hospital. The system combines (1) stage-aware triage, routing high-risk queries to expert templates, (2) hybrid retrieval over curated maternal/newborn guidelines, and (3) evidence-conditioned generation from an LLM. Our core contribution is an evaluation workflow for high-stakes deployment under limited expert supervision. Targeting both component-level and end-to-end testing, we introduce: (i) a labeled triage benchmark (N=150) achieving 86.7% emergency recall, explicitly reporting the missed-emergency vs. over-escalation trade-off; (ii) a synthetic multi-evidence retrieval benchmark (N=100) with chunk-level evidence labels; (iii) LLM-as-judge comparison on real queries (N=781) using clinician-codesigned criteria; and (iv) expert validation. Our findings show that trustworthy medical assistants in multilingual, noisy settings require defense-in-depth design paired with multi-method evaluation, rather than any single model and evaluation method choice. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_13168 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Developing and evaluating a chatbot to support maternal health care Jha, Smriti Jain, Vidhi Xu, Jianyu Liu, Grace Ramesh, Sowmya Nagpal, Jitender Chapman, Gretchen Bellows, Benjamin Goyal, Siddhartha Singh, Aarti Wilder, Bryan Artificial Intelligence Computation and Language Information Retrieval The ability to provide trustworthy maternal health information using phone-based chatbots can have a significant impact, particularly in low-resource settings where users have low health literacy and limited access to care. However, deploying such systems is technically challenging: user queries are short, underspecified, and code-mixed across languages, answers require regional context-specific grounding, and partial or missing symptom context makes safe routing decisions difficult. We present a chatbot for maternal health in India developed through a partnership between academic researchers, a health tech company, a public health nonprofit, and a hospital. The system combines (1) stage-aware triage, routing high-risk queries to expert templates, (2) hybrid retrieval over curated maternal/newborn guidelines, and (3) evidence-conditioned generation from an LLM. Our core contribution is an evaluation workflow for high-stakes deployment under limited expert supervision. Targeting both component-level and end-to-end testing, we introduce: (i) a labeled triage benchmark (N=150) achieving 86.7% emergency recall, explicitly reporting the missed-emergency vs. over-escalation trade-off; (ii) a synthetic multi-evidence retrieval benchmark (N=100) with chunk-level evidence labels; (iii) LLM-as-judge comparison on real queries (N=781) using clinician-codesigned criteria; and (iv) expert validation. Our findings show that trustworthy medical assistants in multilingual, noisy settings require defense-in-depth design paired with multi-method evaluation, rather than any single model and evaluation method choice. |
| title | Developing and evaluating a chatbot to support maternal health care |
| topic | Artificial Intelligence Computation and Language Information Retrieval |
| url | https://arxiv.org/abs/2603.13168 |