Still Not Quite There! Evaluating Large Language Models for Comorbid Mental Health Diagnosis
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866929529313820672 |
|---|---|
| author | Hengle, Amey Kulkarni, Atharva Patankar, Shantanu Chandrasekaran, Madhumitha D'Silva, Sneha Jacob, Jemima Gupta, Rashmi |
| author_facet | Hengle, Amey Kulkarni, Atharva Patankar, Shantanu Chandrasekaran, Madhumitha D'Silva, Sneha Jacob, Jemima Gupta, Rashmi |
| contents | In this study, we introduce ANGST, a novel, first-of-its kind benchmark for depression-anxiety comorbidity classification from social media posts. Unlike contemporary datasets that often oversimplify the intricate interplay between different mental health disorders by treating them as isolated conditions, ANGST enables multi-label classification, allowing each post to be simultaneously identified as indicating depression and/or anxiety. Comprising 2876 meticulously annotated posts by expert psychologists and an additional 7667 silver-labeled posts, ANGST posits a more representative sample of online mental health discourse. Moreover, we benchmark ANGST using various state-of-the-art language models, ranging from Mental-BERT to GPT-4. Our results provide significant insights into the capabilities and limitations of these models in complex diagnostic scenarios. While GPT-4 generally outperforms other models, none achieve an F1 score exceeding 72% in multi-class comorbid classification, underscoring the ongoing challenges in applying language models to mental health diagnostics. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_03908 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Still Not Quite There! Evaluating Large Language Models for Comorbid Mental Health Diagnosis Hengle, Amey Kulkarni, Atharva Patankar, Shantanu Chandrasekaran, Madhumitha D'Silva, Sneha Jacob, Jemima Gupta, Rashmi Computation and Language Artificial Intelligence In this study, we introduce ANGST, a novel, first-of-its kind benchmark for depression-anxiety comorbidity classification from social media posts. Unlike contemporary datasets that often oversimplify the intricate interplay between different mental health disorders by treating them as isolated conditions, ANGST enables multi-label classification, allowing each post to be simultaneously identified as indicating depression and/or anxiety. Comprising 2876 meticulously annotated posts by expert psychologists and an additional 7667 silver-labeled posts, ANGST posits a more representative sample of online mental health discourse. Moreover, we benchmark ANGST using various state-of-the-art language models, ranging from Mental-BERT to GPT-4. Our results provide significant insights into the capabilities and limitations of these models in complex diagnostic scenarios. While GPT-4 generally outperforms other models, none achieve an F1 score exceeding 72% in multi-class comorbid classification, underscoring the ongoing challenges in applying language models to mental health diagnostics. |
| title | Still Not Quite There! Evaluating Large Language Models for Comorbid Mental Health Diagnosis |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2410.03908 |