Towards disentangling the contributions of articulation and acoustics in multimodal phoneme recognition
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912403740950528 |
|---|---|
| author | Foley, Sean Nguyen, Hong Lee, Jihwan Kadiri, Sudarsana Reddy Byrd, Dani Goldstein, Louis Narayanan, Shrikanth |
| author_facet | Foley, Sean Nguyen, Hong Lee, Jihwan Kadiri, Sudarsana Reddy Byrd, Dani Goldstein, Louis Narayanan, Shrikanth |
| contents | Although many previous studies have carried out multimodal learning with real-time MRI data that captures the audio-visual kinematics of the vocal tract during speech, these studies have been limited by their reliance on multi-speaker corpora. This prevents such models from learning a detailed relationship between acoustics and articulation due to considerable cross-speaker variability. In this study, we develop unimodal audio and video models as well as multimodal models for phoneme recognition using a long-form single-speaker MRI corpus, with the goal of disentangling and interpreting the contributions of each modality. Audio and multimodal models show similar performance on different phonetic manner classes but diverge on places of articulation. Interpretation of the models' latent space shows similar encoding of the phonetic space across audio and multimodal models, while the models' attention weights highlight differences in acoustic and articulatory timing for certain phonemes. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_24059 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Towards disentangling the contributions of articulation and acoustics in multimodal phoneme recognition Foley, Sean Nguyen, Hong Lee, Jihwan Kadiri, Sudarsana Reddy Byrd, Dani Goldstein, Louis Narayanan, Shrikanth Machine Learning Although many previous studies have carried out multimodal learning with real-time MRI data that captures the audio-visual kinematics of the vocal tract during speech, these studies have been limited by their reliance on multi-speaker corpora. This prevents such models from learning a detailed relationship between acoustics and articulation due to considerable cross-speaker variability. In this study, we develop unimodal audio and video models as well as multimodal models for phoneme recognition using a long-form single-speaker MRI corpus, with the goal of disentangling and interpreting the contributions of each modality. Audio and multimodal models show similar performance on different phonetic manner classes but diverge on places of articulation. Interpretation of the models' latent space shows similar encoding of the phonetic space across audio and multimodal models, while the models' attention weights highlight differences in acoustic and articulatory timing for certain phonemes. |
| title | Towards disentangling the contributions of articulation and acoustics in multimodal phoneme recognition |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2505.24059 |