Evaluating the Clinical Safety of LLMs in Response to High-Risk Mental Health Disclosures
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909780961918976 |
|---|---|
| author | Shah, Siddharth Gupta, Amit Mann, Aarav Vaz, Alexandre Caldwell, Benjamin E. Scholz, Robert Awad, Peter Allemandi, Rocky Faust, Doug Banka, Harshita Rousmaniere, Tony |
| author_facet | Shah, Siddharth Gupta, Amit Mann, Aarav Vaz, Alexandre Caldwell, Benjamin E. Scholz, Robert Awad, Peter Allemandi, Rocky Faust, Doug Banka, Harshita Rousmaniere, Tony |
| contents | As large language models (LLMs) increasingly mediate emotionally sensitive conversations, especially in mental health contexts, their ability to recognize and respond to high-risk situations becomes a matter of public safety. This study evaluates the responses of six popular LLMs (Claude, Gemini, Deepseek, ChatGPT, Grok 3, and LLAMA) to user prompts simulating crisis-level mental health disclosures. Drawing on a coding framework developed by licensed clinicians, five safety-oriented behaviors were assessed: explicit risk acknowledgment, empathy, encouragement to seek help, provision of specific resources, and invitation to continue the conversation. Claude outperformed all others in global assessment, while Grok 3, ChatGPT, and LLAMA underperformed across multiple domains. Notably, most models exhibited empathy, but few consistently provided practical support or sustained engagement. These findings suggest that while LLMs show potential for emotionally attuned communication, none currently meet satisfactory clinical standards for crisis response. Ongoing development and targeted fine-tuning are essential to ensure ethical deployment of AI in mental health settings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_08839 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Evaluating the Clinical Safety of LLMs in Response to High-Risk Mental Health Disclosures Shah, Siddharth Gupta, Amit Mann, Aarav Vaz, Alexandre Caldwell, Benjamin E. Scholz, Robert Awad, Peter Allemandi, Rocky Faust, Doug Banka, Harshita Rousmaniere, Tony Computers and Society I.2.7 As large language models (LLMs) increasingly mediate emotionally sensitive conversations, especially in mental health contexts, their ability to recognize and respond to high-risk situations becomes a matter of public safety. This study evaluates the responses of six popular LLMs (Claude, Gemini, Deepseek, ChatGPT, Grok 3, and LLAMA) to user prompts simulating crisis-level mental health disclosures. Drawing on a coding framework developed by licensed clinicians, five safety-oriented behaviors were assessed: explicit risk acknowledgment, empathy, encouragement to seek help, provision of specific resources, and invitation to continue the conversation. Claude outperformed all others in global assessment, while Grok 3, ChatGPT, and LLAMA underperformed across multiple domains. Notably, most models exhibited empathy, but few consistently provided practical support or sustained engagement. These findings suggest that while LLMs show potential for emotionally attuned communication, none currently meet satisfactory clinical standards for crisis response. Ongoing development and targeted fine-tuning are essential to ensure ethical deployment of AI in mental health settings. |
| title | Evaluating the Clinical Safety of LLMs in Response to High-Risk Mental Health Disclosures |
| topic | Computers and Society I.2.7 |
| url | https://arxiv.org/abs/2509.08839 |