Towards Child-Inclusive Clinical Video Understanding for Autism Spectrum Disorder
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929509003952128 |
|---|---|
| author | Kommineni, Aditya Bose, Digbalay Feng, Tiantian Kim, So Hyun Tager-Flusberg, Helen Bishop, Somer Lord, Catherine Kadiri, Sudarsana Narayanan, Shrikanth |
| author_facet | Kommineni, Aditya Bose, Digbalay Feng, Tiantian Kim, So Hyun Tager-Flusberg, Helen Bishop, Somer Lord, Catherine Kadiri, Sudarsana Narayanan, Shrikanth |
| contents | Clinical videos in the context of Autism Spectrum Disorder are often long-form interactions between children and caregivers/clinical professionals, encompassing complex verbal and non-verbal behaviors. Objective analyses of these videos could provide clinicians and researchers with nuanced insights into the behavior of children with Autism Spectrum Disorder. Manually coding these videos is a time-consuming task and requires a high level of domain expertise. Hence, the ability to capture these interactions computationally can augment the manual effort and enable supporting the diagnostic procedure. In this work, we investigate the use of foundation models across three modalities: speech, video, and text, to analyse child-focused interaction sessions. We propose a unified methodology to combine multiple modalities by using large language models as reasoning agents. We evaluate their performance on two tasks with different information granularity: activity recognition and abnormal behavior detection. We find that the proposed multimodal pipeline provides robustness to modality-specific limitations and improves performance on the clinical video analysis compared to unimodal settings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_13606 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Towards Child-Inclusive Clinical Video Understanding for Autism Spectrum Disorder Kommineni, Aditya Bose, Digbalay Feng, Tiantian Kim, So Hyun Tager-Flusberg, Helen Bishop, Somer Lord, Catherine Kadiri, Sudarsana Narayanan, Shrikanth Computer Vision and Pattern Recognition Machine Learning Clinical videos in the context of Autism Spectrum Disorder are often long-form interactions between children and caregivers/clinical professionals, encompassing complex verbal and non-verbal behaviors. Objective analyses of these videos could provide clinicians and researchers with nuanced insights into the behavior of children with Autism Spectrum Disorder. Manually coding these videos is a time-consuming task and requires a high level of domain expertise. Hence, the ability to capture these interactions computationally can augment the manual effort and enable supporting the diagnostic procedure. In this work, we investigate the use of foundation models across three modalities: speech, video, and text, to analyse child-focused interaction sessions. We propose a unified methodology to combine multiple modalities by using large language models as reasoning agents. We evaluate their performance on two tasks with different information granularity: activity recognition and abnormal behavior detection. We find that the proposed multimodal pipeline provides robustness to modality-specific limitations and improves performance on the clinical video analysis compared to unimodal settings. |
| title | Towards Child-Inclusive Clinical Video Understanding for Autism Spectrum Disorder |
| topic | Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2409.13606 |