Reading Between the Frames: Multi-Modal Depression Detection in Videos from Non-Verbal Cues
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866929199808249856 |
|---|---|
| author | Gimeno-Gómez, David Bucur, Ana-Maria Cosma, Adrian Martínez-Hinarejos, Carlos-David Rosso, Paolo |
| author_facet | Gimeno-Gómez, David Bucur, Ana-Maria Cosma, Adrian Martínez-Hinarejos, Carlos-David Rosso, Paolo |
| contents | Depression, a prominent contributor to global disability, affects a substantial portion of the population. Efforts to detect depression from social media texts have been prevalent, yet only a few works explored depression detection from user-generated video content. In this work, we address this research gap by proposing a simple and flexible multi-modal temporal model capable of discerning non-verbal depression cues from diverse modalities in noisy, real-world videos. We show that, for in-the-wild videos, using additional high-level non-verbal cues is crucial to achieving good performance, and we extracted and processed audio speech embeddings, face emotion embeddings, face, body and hand landmarks, and gaze and blinking information. Through extensive experiments, we show that our model achieves state-of-the-art results on three key benchmark datasets for depression detection from video by a substantial margin. Our code is publicly available on GitHub. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_02746 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Reading Between the Frames: Multi-Modal Depression Detection in Videos from Non-Verbal Cues Gimeno-Gómez, David Bucur, Ana-Maria Cosma, Adrian Martínez-Hinarejos, Carlos-David Rosso, Paolo Computer Vision and Pattern Recognition Depression, a prominent contributor to global disability, affects a substantial portion of the population. Efforts to detect depression from social media texts have been prevalent, yet only a few works explored depression detection from user-generated video content. In this work, we address this research gap by proposing a simple and flexible multi-modal temporal model capable of discerning non-verbal depression cues from diverse modalities in noisy, real-world videos. We show that, for in-the-wild videos, using additional high-level non-verbal cues is crucial to achieving good performance, and we extracted and processed audio speech embeddings, face emotion embeddings, face, body and hand landmarks, and gaze and blinking information. Through extensive experiments, we show that our model achieves state-of-the-art results on three key benchmark datasets for depression detection from video by a substantial margin. Our code is publicly available on GitHub. |
| title | Reading Between the Frames: Multi-Modal Depression Detection in Videos from Non-Verbal Cues |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2401.02746 |