Saved in:
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2605.18466 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914578347065344 |
|---|---|
| author | Liu, Daiqi Mulzer, Lukas Hasan, Md de Castro, Nyvenn Xing, Fangxu Kang, Xingjian Ye, Chengze Mei, Siyuan Sun, Yipeng Arias-Vergara, Tomás Hutter, Jana Woo, Jonghye Maier, Andreas Pérez-Toro, Paula Andrea |
| author_facet | Liu, Daiqi Mulzer, Lukas Hasan, Md de Castro, Nyvenn Xing, Fangxu Kang, Xingjian Ye, Chengze Mei, Siyuan Sun, Yipeng Arias-Vergara, Tomás Hutter, Jana Woo, Jonghye Maier, Andreas Pérez-Toro, Paula Andrea |
| contents | Segmenting vocal tract articulators in real-time MRI (rtMRI) is a challenging dynamic image segmentation problem characterized by low contrast, rapid motion, and limited spatial resolution. However, while rtMRI acquisitions may provide synchronized acoustic signals, existing methods discard this information, and the few multimodal approaches that incorporate audio cannot be deployed when audio is unavailable. We propose a three-stage framework that leverages acoustic and phonological supervision during training while requiring only the rtMRI image at inference: phonological representations are converted into spatial bounding-box priors for articulator localization, visual and acoustic encoders are aligned via dual-level cross-modal contrastive pretraining, and the learned representations are fused through a cross-attention decoder, effectively transferring multimodal knowledge into a single-modality inference pipeline. Evaluated on 75-Speaker~Annot-16 and USC-TIMIT datasets, our method outperforms existing unimodal and multimodal methods, demonstrating that multimodal supervision provides transferable benefits for precise and clinically deployable vocal tract segmentation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_18466 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Speech-Guided Multimodal Learning for Vocal Tract Segmentation in Real-Time MRI Liu, Daiqi Mulzer, Lukas Hasan, Md de Castro, Nyvenn Xing, Fangxu Kang, Xingjian Ye, Chengze Mei, Siyuan Sun, Yipeng Arias-Vergara, Tomás Hutter, Jana Woo, Jonghye Maier, Andreas Pérez-Toro, Paula Andrea Computer Vision and Pattern Recognition Segmenting vocal tract articulators in real-time MRI (rtMRI) is a challenging dynamic image segmentation problem characterized by low contrast, rapid motion, and limited spatial resolution. However, while rtMRI acquisitions may provide synchronized acoustic signals, existing methods discard this information, and the few multimodal approaches that incorporate audio cannot be deployed when audio is unavailable. We propose a three-stage framework that leverages acoustic and phonological supervision during training while requiring only the rtMRI image at inference: phonological representations are converted into spatial bounding-box priors for articulator localization, visual and acoustic encoders are aligned via dual-level cross-modal contrastive pretraining, and the learned representations are fused through a cross-attention decoder, effectively transferring multimodal knowledge into a single-modality inference pipeline. Evaluated on 75-Speaker~Annot-16 and USC-TIMIT datasets, our method outperforms existing unimodal and multimodal methods, demonstrating that multimodal supervision provides transferable benefits for precise and clinically deployable vocal tract segmentation. |
| title | Speech-Guided Multimodal Learning for Vocal Tract Segmentation in Real-Time MRI |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2605.18466 |