A Multimodal Framework for the Assessment of the Schizophrenia Spectrum
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909315889102848 |
|---|---|
| author | Premananth, Gowtham Siriwardena, Yashish M. Resnik, Philip Bansal, Sonia Kelly, Deanna L. Espy-Wilson, Carol |
| author_facet | Premananth, Gowtham Siriwardena, Yashish M. Resnik, Philip Bansal, Sonia Kelly, Deanna L. Espy-Wilson, Carol |
| contents | This paper presents a novel multimodal framework to distinguish between different symptom classes of subjects in the schizophrenia spectrum and healthy controls using audio, video, and text modalities. We implemented Convolution Neural Network and Long Short Term Memory based unimodal models and experimented on various multimodal fusion approaches to come up with the proposed framework. We utilized a minimal Gated multimodal unit (mGMU) to obtain a bi-modal intermediate fusion of the features extracted from the input modalities before finally fusing the outputs of the bimodal fusions to perform subject-wise classifications. The use of mGMU units in the multimodal framework improved the performance in both weighted f1-score and weighted AUC-ROC scores. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_09706 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | A Multimodal Framework for the Assessment of the Schizophrenia Spectrum Premananth, Gowtham Siriwardena, Yashish M. Resnik, Philip Bansal, Sonia Kelly, Deanna L. Espy-Wilson, Carol Audio and Speech Processing This paper presents a novel multimodal framework to distinguish between different symptom classes of subjects in the schizophrenia spectrum and healthy controls using audio, video, and text modalities. We implemented Convolution Neural Network and Long Short Term Memory based unimodal models and experimented on various multimodal fusion approaches to come up with the proposed framework. We utilized a minimal Gated multimodal unit (mGMU) to obtain a bi-modal intermediate fusion of the features extracted from the input modalities before finally fusing the outputs of the bimodal fusions to perform subject-wise classifications. The use of mGMU units in the multimodal framework improved the performance in both weighted f1-score and weighted AUC-ROC scores. |
| title | A Multimodal Framework for the Assessment of the Schizophrenia Spectrum |
| topic | Audio and Speech Processing |
| url | https://arxiv.org/abs/2406.09706 |