PsyEval: A Suite of Mental Health Related Tasks for Evaluating Large Language Models
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2023
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909215003508736 |
|---|---|
| author | Jin, Haoan Chen, Siyuan Dilixiati, Dilawaier Jiang, Yewei Wu, Mengyue Zhu, Kenny Q. |
| author_facet | Jin, Haoan Chen, Siyuan Dilixiati, Dilawaier Jiang, Yewei Wu, Mengyue Zhu, Kenny Q. |
| contents | Evaluating Large Language Models (LLMs) in the mental health domain poses distinct challenged from other domains, given the subtle and highly subjective nature of symptoms that exhibit significant variability among individuals. This paper presents PsyEval, the first comprehensive suite of mental health-related tasks for evaluating LLMs. PsyEval encompasses five sub-tasks that evaluate three critical dimensions of mental health. This comprehensive framework is designed to thoroughly assess the unique challenges and intricacies of mental health-related tasks, making PsyEval a highly specialized and valuable tool for evaluating LLM performance in this domain. We evaluate twelve advanced LLMs using PsyEval. Experiment results not only demonstrate significant room for improvement in current LLMs concerning mental health but also unveil potential directions for future model optimization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2311_09189 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | PsyEval: A Suite of Mental Health Related Tasks for Evaluating Large Language Models Jin, Haoan Chen, Siyuan Dilixiati, Dilawaier Jiang, Yewei Wu, Mengyue Zhu, Kenny Q. Computation and Language Evaluating Large Language Models (LLMs) in the mental health domain poses distinct challenged from other domains, given the subtle and highly subjective nature of symptoms that exhibit significant variability among individuals. This paper presents PsyEval, the first comprehensive suite of mental health-related tasks for evaluating LLMs. PsyEval encompasses five sub-tasks that evaluate three critical dimensions of mental health. This comprehensive framework is designed to thoroughly assess the unique challenges and intricacies of mental health-related tasks, making PsyEval a highly specialized and valuable tool for evaluating LLM performance in this domain. We evaluate twelve advanced LLMs using PsyEval. Experiment results not only demonstrate significant room for improvement in current LLMs concerning mental health but also unveil potential directions for future model optimization. |
| title | PsyEval: A Suite of Mental Health Related Tasks for Evaluating Large Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2311.09189 |