Joint Speaker Features Learning for Audio-visual Multichannel Speech Separation and Recognition
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917694253563904 |
|---|---|
| author | Li, Guinan Deng, Jiajun Chen, Youjun Geng, Mengzhe Hu, Shujie Li, Zhe Jin, Zengrui Wang, Tianzi Xie, Xurong Meng, Helen Liu, Xunying |
| author_facet | Li, Guinan Deng, Jiajun Chen, Youjun Geng, Mengzhe Hu, Shujie Li, Zhe Jin, Zengrui Wang, Tianzi Xie, Xurong Meng, Helen Liu, Xunying |
| contents | This paper proposes joint speaker feature learning methods for zero-shot adaptation of audio-visual multichannel speech separation and recognition systems. xVector and ECAPA-TDNN speaker encoders are connected using purpose-built fusion blocks and tightly integrated with the complete system training. Experiments conducted on LRS3-TED data simulated multichannel overlapped speech suggest that joint speaker feature learning consistently improves speech separation and recognition performance over the baselines without joint speaker feature estimation. Further analyses reveal performance improvements are strongly correlated with increased inter-speaker discrimination measured using cosine similarity. The best-performing joint speaker feature learning adapted system outperformed the baseline fine-tuned WavLM model by statistically significant WER reductions of 21.6% and 25.3% absolute (67.5% and 83.5% relative) on Dev and Test sets after incorporating WavLM features and video modality. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_10152 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Joint Speaker Features Learning for Audio-visual Multichannel Speech Separation and Recognition Li, Guinan Deng, Jiajun Chen, Youjun Geng, Mengzhe Hu, Shujie Li, Zhe Jin, Zengrui Wang, Tianzi Xie, Xurong Meng, Helen Liu, Xunying Sound Audio and Speech Processing This paper proposes joint speaker feature learning methods for zero-shot adaptation of audio-visual multichannel speech separation and recognition systems. xVector and ECAPA-TDNN speaker encoders are connected using purpose-built fusion blocks and tightly integrated with the complete system training. Experiments conducted on LRS3-TED data simulated multichannel overlapped speech suggest that joint speaker feature learning consistently improves speech separation and recognition performance over the baselines without joint speaker feature estimation. Further analyses reveal performance improvements are strongly correlated with increased inter-speaker discrimination measured using cosine similarity. The best-performing joint speaker feature learning adapted system outperformed the baseline fine-tuned WavLM model by statistically significant WER reductions of 21.6% and 25.3% absolute (67.5% and 83.5% relative) on Dev and Test sets after incorporating WavLM features and video modality. |
| title | Joint Speaker Features Learning for Audio-visual Multichannel Speech Separation and Recognition |
| topic | Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2406.10152 |