Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2406.02554 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916274326470656 |
|---|---|
| author | Deng, Shijian Kosloski, Erin E. Patel, Siddhi Barnett, Zeke A. Nan, Yiyang Kaplan, Alexander Aarukapalli, Sisira Doan, William T. Wang, Matthew Singh, Harsh Rollins, Pamela R. Tian, Yapeng |
| author_facet | Deng, Shijian Kosloski, Erin E. Patel, Siddhi Barnett, Zeke A. Nan, Yiyang Kaplan, Alexander Aarukapalli, Sisira Doan, William T. Wang, Matthew Singh, Harsh Rollins, Pamela R. Tian, Yapeng |
| contents | In this article, we introduce a novel problem of audio-visual autism behavior recognition, which includes social behavior recognition, an essential aspect previously omitted in AI-assisted autism screening research. We define the task at hand as one that is audio-visual autism behavior recognition, which uses audio and visual cues, including any speech present in the audio, to recognize autism-related behaviors. To facilitate this new research direction, we collected an audio-visual autism spectrum dataset (AV-ASD), currently the largest video dataset for autism screening using a behavioral approach. It covers an extensive range of autism-associated behaviors, including those related to social communication and interaction. To pave the way for further research on this new problem, we intensively explored leveraging foundation models and multimodal large language models across different modalities. Our experiments on the AV-ASD dataset demonstrate that integrating audio, visual, and speech modalities significantly enhances the performance in autism behavior recognition. Additionally, we explored the use of a post-hoc to ad-hoc pipeline in a multimodal large language model to investigate its potential to augment the model's explanatory capability during autism behavior recognition. We will release our dataset, code, and pre-trained models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_02554 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Hear Me, See Me, Understand Me: Audio-Visual Autism Behavior Recognition Deng, Shijian Kosloski, Erin E. Patel, Siddhi Barnett, Zeke A. Nan, Yiyang Kaplan, Alexander Aarukapalli, Sisira Doan, William T. Wang, Matthew Singh, Harsh Rollins, Pamela R. Tian, Yapeng Audio and Speech Processing Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition Machine Learning Multimedia In this article, we introduce a novel problem of audio-visual autism behavior recognition, which includes social behavior recognition, an essential aspect previously omitted in AI-assisted autism screening research. We define the task at hand as one that is audio-visual autism behavior recognition, which uses audio and visual cues, including any speech present in the audio, to recognize autism-related behaviors. To facilitate this new research direction, we collected an audio-visual autism spectrum dataset (AV-ASD), currently the largest video dataset for autism screening using a behavioral approach. It covers an extensive range of autism-associated behaviors, including those related to social communication and interaction. To pave the way for further research on this new problem, we intensively explored leveraging foundation models and multimodal large language models across different modalities. Our experiments on the AV-ASD dataset demonstrate that integrating audio, visual, and speech modalities significantly enhances the performance in autism behavior recognition. Additionally, we explored the use of a post-hoc to ad-hoc pipeline in a multimodal large language model to investigate its potential to augment the model's explanatory capability during autism behavior recognition. We will release our dataset, code, and pre-trained models. |
| title | Hear Me, See Me, Understand Me: Audio-Visual Autism Behavior Recognition |
| topic | Audio and Speech Processing Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition Machine Learning Multimedia |
| url | https://arxiv.org/abs/2406.02554 |