Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2502.17899 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918284894404608 |
|---|---|
| author | Li, Tong Yang, Shu Wu, Junchao Wei, Jiyao Hu, Lijie Li, Mengdi Wong, Derek F. Oltmanns, Joshua R. Wang, Di |
| author_facet | Li, Tong Yang, Shu Wu, Junchao Wei, Jiyao Hu, Lijie Li, Mengdi Wong, Derek F. Oltmanns, Joshua R. Wang, Di |
| contents | We present a comprehensive evaluation framework for assessing Large Language Models' (LLMs) capabilities in suicide prevention, focusing on two critical aspects: the Identification of Implicit Suicidal ideation (IIS) and the Provision of Appropriate Supportive responses (PAS). We introduce \ourdata, a novel dataset of 1,308 test cases built upon psychological frameworks including D/S-IAT and Negative Automatic Thinking, alongside real-world scenarios. Through extensive experiments with 8 widely used LLMs under different contextual settings, we find that current models struggle significantly with detecting implicit suicidal ideation and providing appropriate support, highlighting crucial limitations in applying LLMs to mental health contexts. Our findings underscore the need for more sophisticated approaches in developing and evaluating LLMs for sensitive psychological applications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_17899 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Can Large Language Models Identify Implicit Suicidal Ideation? An Empirical Evaluation Li, Tong Yang, Shu Wu, Junchao Wei, Jiyao Hu, Lijie Li, Mengdi Wong, Derek F. Oltmanns, Joshua R. Wang, Di Computation and Language We present a comprehensive evaluation framework for assessing Large Language Models' (LLMs) capabilities in suicide prevention, focusing on two critical aspects: the Identification of Implicit Suicidal ideation (IIS) and the Provision of Appropriate Supportive responses (PAS). We introduce \ourdata, a novel dataset of 1,308 test cases built upon psychological frameworks including D/S-IAT and Negative Automatic Thinking, alongside real-world scenarios. Through extensive experiments with 8 widely used LLMs under different contextual settings, we find that current models struggle significantly with detecting implicit suicidal ideation and providing appropriate support, highlighting crucial limitations in applying LLMs to mental health contexts. Our findings underscore the need for more sophisticated approaches in developing and evaluating LLMs for sensitive psychological applications. |
| title | Can Large Language Models Identify Implicit Suicidal Ideation? An Empirical Evaluation |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2502.17899 |