Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2603.29429 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912990706532352 |
|---|---|
| author | Li, Yahan Du, Chaohao Li, Zeyang Kuizon, Christopher Chun Cheng, Shupeng Hwang, Angel Hsing-Chi Frank, Adam C. Liu, Ruishan |
| author_facet | Li, Yahan Du, Chaohao Li, Zeyang Kuizon, Christopher Chun Cheng, Shupeng Hwang, Angel Hsing-Chi Frank, Adam C. Liu, Ruishan |
| contents | Mental-health support is increasingly mediated by conversational systems (e.g., LLM-based tools), but users often lack structured ways to audit the quality and potential risks of the support they receive. We introduce CounselReflect, an end-to-end toolkit for auditing mental-health support dialogues. Rather than producing a single opaque quality score, CounselReflect provides structured, multi-dimensional reports with session-level summaries, turn-level scores, and evidence-linked excerpts to support transparent inspection. The system integrates two families of evaluation signals: (i) 12 model-based metrics produced by task-specific predictors, and (ii) rubric-based metrics that extend coverage via a literature-derived library (69 metrics) and user-defined custom metrics, operationalized with configurable LLM judges. CounselReflect is available as a web application, browser extension, and command-line interface (CLI), enabling use in real-time settings as well as at scale. Human evaluation includes a user study with 20 participants and an expert review with 6 mental-health professionals, suggesting that CounselReflect supports understandable, usable, and trustworthy auditing. A demo video and full source code are also provided. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_29429 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | CounselReflect: A Toolkit for Auditing Mental-Health Dialogues Li, Yahan Du, Chaohao Li, Zeyang Kuizon, Christopher Chun Cheng, Shupeng Hwang, Angel Hsing-Chi Frank, Adam C. Liu, Ruishan Computation and Language Mental-health support is increasingly mediated by conversational systems (e.g., LLM-based tools), but users often lack structured ways to audit the quality and potential risks of the support they receive. We introduce CounselReflect, an end-to-end toolkit for auditing mental-health support dialogues. Rather than producing a single opaque quality score, CounselReflect provides structured, multi-dimensional reports with session-level summaries, turn-level scores, and evidence-linked excerpts to support transparent inspection. The system integrates two families of evaluation signals: (i) 12 model-based metrics produced by task-specific predictors, and (ii) rubric-based metrics that extend coverage via a literature-derived library (69 metrics) and user-defined custom metrics, operationalized with configurable LLM judges. CounselReflect is available as a web application, browser extension, and command-line interface (CLI), enabling use in real-time settings as well as at scale. Human evaluation includes a user study with 20 participants and an expert review with 6 mental-health professionals, suggesting that CounselReflect supports understandable, usable, and trustworthy auditing. A demo video and full source code are also provided. |
| title | CounselReflect: A Toolkit for Auditing Mental-Health Dialogues |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2603.29429 |