Gesture-Aware Zero-Shot Speech Recognition for Patients with Language Disorders
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916621363183616 |
|---|---|
| author | Kim, Seungbae Lee, Daeun Stark, Brielle Han, Jinyoung |
| author_facet | Kim, Seungbae Lee, Daeun Stark, Brielle Han, Jinyoung |
| contents | Individuals with language disorders often face significant communication challenges due to their limited language processing and comprehension abilities, which also affect their interactions with voice-assisted systems that mostly rely on Automatic Speech Recognition (ASR). Despite advancements in ASR that address disfluencies, there has been little attention on integrating non-verbal communication methods, such as gestures, which individuals with language disorders substantially rely on to supplement their communication. Recognizing the need to interpret the latent meanings of visual information not captured by speech alone, we propose a gesture-aware ASR system utilizing a multimodal large language model with zero-shot learning for individuals with speech impairments. Our experiment results and analyses show that including gesture information significantly enhances semantic understanding. This study can help develop effective communication technologies, specifically designed to meet the unique needs of individuals with language impairments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_13983 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Gesture-Aware Zero-Shot Speech Recognition for Patients with Language Disorders Kim, Seungbae Lee, Daeun Stark, Brielle Han, Jinyoung Audio and Speech Processing Artificial Intelligence Individuals with language disorders often face significant communication challenges due to their limited language processing and comprehension abilities, which also affect their interactions with voice-assisted systems that mostly rely on Automatic Speech Recognition (ASR). Despite advancements in ASR that address disfluencies, there has been little attention on integrating non-verbal communication methods, such as gestures, which individuals with language disorders substantially rely on to supplement their communication. Recognizing the need to interpret the latent meanings of visual information not captured by speech alone, we propose a gesture-aware ASR system utilizing a multimodal large language model with zero-shot learning for individuals with speech impairments. Our experiment results and analyses show that including gesture information significantly enhances semantic understanding. This study can help develop effective communication technologies, specifically designed to meet the unique needs of individuals with language impairments. |
| title | Gesture-Aware Zero-Shot Speech Recognition for Patients with Language Disorders |
| topic | Audio and Speech Processing Artificial Intelligence |
| url | https://arxiv.org/abs/2502.13983 |