Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2510.13812 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911213400621056 |
|---|---|
| author | Dwyer, Bridget Flathers, Matthew Sano, Akane Dempsey, Allison Cipriani, Andrea Gazi, Asim H. Gorban, Carla Rodriguez, Carolyn I. Stromeyer IV, Charles King, Darlene Rozenblit, Eden Strudwick, Gillian Linardon, Jake Cheong, Jiaee Firth, Joseph Herpertz, Julian Schwarz, Julian Emerson, Margaret Paulus, Martin P. Patriquin, Michelle Hua, Yining Choudhary, Soumya Siddals, Steven Pinillos, Laura Ospina Bantjes, Jason Scheuller, Steven Xu, Xuhai Duckworth, Ken Gillison, Daniel H. Wood, Michael Torous, John |
| author_facet | Dwyer, Bridget Flathers, Matthew Sano, Akane Dempsey, Allison Cipriani, Andrea Gazi, Asim H. Gorban, Carla Rodriguez, Carolyn I. Stromeyer IV, Charles King, Darlene Rozenblit, Eden Strudwick, Gillian Linardon, Jake Cheong, Jiaee Firth, Joseph Herpertz, Julian Schwarz, Julian Emerson, Margaret Paulus, Martin P. Patriquin, Michelle Hua, Yining Choudhary, Soumya Siddals, Steven Pinillos, Laura Ospina Bantjes, Jason Scheuller, Steven Xu, Xuhai Duckworth, Ken Gillison, Daniel H. Wood, Michael Torous, John |
| contents | Individuals are increasingly utilizing large language model (LLM)based tools for mental health guidance and crisis support in place of human experts. While AI technology has great potential to improve health outcomes, insufficient empirical evidence exists to suggest that AI technology can be deployed as a clinical replacement; thus, there is an urgent need to assess and regulate such tools. Regulatory efforts have been made and multiple evaluation frameworks have been proposed, however,field-wide assessment metrics have yet to be formally integrated. In this paper, we introduce a comprehensive online platform that aggregates evaluation approaches and serves as a dynamic online resource to simplify LLM and LLM-based tool assessment: MindBenchAI. At its core, MindBenchAI is designed to provide easily accessible/interpretable information for diverse stakeholders (patients, clinicians, developers, regulators, etc.). To create MindBenchAI, we built off our work developing MINDapps.org to support informed decision-making around smartphone app use for mental health, and expanded the technical MINDapps.org framework to encompass novel large language model (LLM) functionalities through benchmarking approaches. The MindBenchAI platform is designed as a partnership with the National Alliance on Mental Illness (NAMI) to provide assessment tools that systematically evaluate LLMs and LLM-based tools with objective and transparent criteria from a healthcare standpoint, assessing both profile (i.e. technical features, privacy protections, and conversational style) and performance characteristics (i.e. clinical reasoning skills). |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_13812 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MindBenchAI: An Actionable Platform to Evaluate the Profile and Performance of Large Language Models in a Mental Healthcare Context Dwyer, Bridget Flathers, Matthew Sano, Akane Dempsey, Allison Cipriani, Andrea Gazi, Asim H. Gorban, Carla Rodriguez, Carolyn I. Stromeyer IV, Charles King, Darlene Rozenblit, Eden Strudwick, Gillian Linardon, Jake Cheong, Jiaee Firth, Joseph Herpertz, Julian Schwarz, Julian Emerson, Margaret Paulus, Martin P. Patriquin, Michelle Hua, Yining Choudhary, Soumya Siddals, Steven Pinillos, Laura Ospina Bantjes, Jason Scheuller, Steven Xu, Xuhai Duckworth, Ken Gillison, Daniel H. Wood, Michael Torous, John Human-Computer Interaction Individuals are increasingly utilizing large language model (LLM)based tools for mental health guidance and crisis support in place of human experts. While AI technology has great potential to improve health outcomes, insufficient empirical evidence exists to suggest that AI technology can be deployed as a clinical replacement; thus, there is an urgent need to assess and regulate such tools. Regulatory efforts have been made and multiple evaluation frameworks have been proposed, however,field-wide assessment metrics have yet to be formally integrated. In this paper, we introduce a comprehensive online platform that aggregates evaluation approaches and serves as a dynamic online resource to simplify LLM and LLM-based tool assessment: MindBenchAI. At its core, MindBenchAI is designed to provide easily accessible/interpretable information for diverse stakeholders (patients, clinicians, developers, regulators, etc.). To create MindBenchAI, we built off our work developing MINDapps.org to support informed decision-making around smartphone app use for mental health, and expanded the technical MINDapps.org framework to encompass novel large language model (LLM) functionalities through benchmarking approaches. The MindBenchAI platform is designed as a partnership with the National Alliance on Mental Illness (NAMI) to provide assessment tools that systematically evaluate LLMs and LLM-based tools with objective and transparent criteria from a healthcare standpoint, assessing both profile (i.e. technical features, privacy protections, and conversational style) and performance characteristics (i.e. clinical reasoning skills). |
| title | MindBenchAI: An Actionable Platform to Evaluate the Profile and Performance of Large Language Models in a Mental Healthcare Context |
| topic | Human-Computer Interaction |
| url | https://arxiv.org/abs/2510.13812 |