Multimodal Segmentation for Vocal Tract Modeling
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910499473457152 |
|---|---|
| author | Jain, Rishi Yu, Bohan Wu, Peter Prabhune, Tejas Anumanchipalli, Gopala |
| author_facet | Jain, Rishi Yu, Bohan Wu, Peter Prabhune, Tejas Anumanchipalli, Gopala |
| contents | Accurate modeling of the vocal tract is necessary to construct articulatory representations for interpretable speech processing and linguistics. However, vocal tract modeling is challenging because many internal articulators are occluded from external motion capture technologies. Real-time magnetic resonance imaging (RT-MRI) allows measuring precise movements of internal articulators during speech, but annotated datasets of MRI are limited in size due to time-consuming and computationally expensive labeling methods. We first present a deep labeling strategy for the RT-MRI video using a vision-only segmentation approach. We then introduce a multimodal algorithm using audio to improve segmentation of vocal articulators. Together, we set a new benchmark for vocal tract modeling in MRI video segmentation and use this to release labels for a 75-speaker RT-MRI dataset, increasing the amount of labeled public RT-MRI data of the vocal tract by over a factor of 9. The code and dataset labels can be found at \url{rishiraij.github.io/multimodal-mri-avatar/}. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_15754 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Multimodal Segmentation for Vocal Tract Modeling Jain, Rishi Yu, Bohan Wu, Peter Prabhune, Tejas Anumanchipalli, Gopala Computer Vision and Pattern Recognition Computation and Language Machine Learning Sound Audio and Speech Processing Accurate modeling of the vocal tract is necessary to construct articulatory representations for interpretable speech processing and linguistics. However, vocal tract modeling is challenging because many internal articulators are occluded from external motion capture technologies. Real-time magnetic resonance imaging (RT-MRI) allows measuring precise movements of internal articulators during speech, but annotated datasets of MRI are limited in size due to time-consuming and computationally expensive labeling methods. We first present a deep labeling strategy for the RT-MRI video using a vision-only segmentation approach. We then introduce a multimodal algorithm using audio to improve segmentation of vocal articulators. Together, we set a new benchmark for vocal tract modeling in MRI video segmentation and use this to release labels for a 75-speaker RT-MRI dataset, increasing the amount of labeled public RT-MRI data of the vocal tract by over a factor of 9. The code and dataset labels can be found at \url{rishiraij.github.io/multimodal-mri-avatar/}. |
| title | Multimodal Segmentation for Vocal Tract Modeling |
| topic | Computer Vision and Pattern Recognition Computation and Language Machine Learning Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2406.15754 |