Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2406.15362 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916439540105216 |
|---|---|
| author | Ali, Nuredin Zhang, Charles Chuankai Mayo, Ned Chancellor, Stevie |
| author_facet | Ali, Nuredin Zhang, Charles Chuankai Mayo, Ned Chancellor, Stevie |
| contents | Social media data has been used for detecting users with mental disorders, such as depression. Despite the global significance of cross-cultural representation and its potential impact on model performance, publicly available datasets often lack crucial metadata related to this aspect. In this work, we evaluate the generalization of benchmark datasets to build AI models on cross-cultural Twitter data. We gather a custom geo-located Twitter dataset of depressed users from seven countries as a test dataset. Our results show that depression detection models do not generalize globally. The models perform worse on Global South users compared to Global North. Pre-trained language models achieve the best generalization compared to Logistic Regression, though still show significant gaps in performance on depressed and non-Western users. We quantify our findings and provide several actionable suggestions to mitigate this issue. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_15362 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Diverse Perspectives, Divergent Models: Cross-Cultural Evaluation of Depression Detection on Twitter Ali, Nuredin Zhang, Charles Chuankai Mayo, Ned Chancellor, Stevie Computation and Language Social media data has been used for detecting users with mental disorders, such as depression. Despite the global significance of cross-cultural representation and its potential impact on model performance, publicly available datasets often lack crucial metadata related to this aspect. In this work, we evaluate the generalization of benchmark datasets to build AI models on cross-cultural Twitter data. We gather a custom geo-located Twitter dataset of depressed users from seven countries as a test dataset. Our results show that depression detection models do not generalize globally. The models perform worse on Global South users compared to Global North. Pre-trained language models achieve the best generalization compared to Logistic Regression, though still show significant gaps in performance on depressed and non-Western users. We quantify our findings and provide several actionable suggestions to mitigate this issue. |
| title | Diverse Perspectives, Divergent Models: Cross-Cultural Evaluation of Depression Detection on Twitter |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2406.15362 |