Empowering Whisper as a Joint Multi-Talker and Target-Talker Speech Recognition System
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910575567568896 |
|---|---|
| author | Meng, Lingwei Kang, Jiawen Wang, Yuejiao Jin, Zengrui Wu, Xixin Liu, Xunying Meng, Helen |
| author_facet | Meng, Lingwei Kang, Jiawen Wang, Yuejiao Jin, Zengrui Wu, Xixin Liu, Xunying Meng, Helen |
| contents | Multi-talker speech recognition and target-talker speech recognition, both involve transcription in multi-talker contexts, remain significant challenges. However, existing methods rarely attempt to simultaneously address both tasks. In this study, we propose a pioneering approach to empower Whisper, which is a speech foundation model, to tackle joint multi-talker and target-talker speech recognition tasks. Specifically, (i) we freeze Whisper and plug a Sidecar separator into its encoder to separate mixed embedding for multiple talkers; (ii) a Target Talker Identifier is introduced to identify the embedding flow of the target talker on the fly, requiring only three-second enrollment speech as a cue; (iii) soft prompt tuning for decoder is explored for better task adaptation. Our method outperforms previous methods on two- and three-talker LibriMix and LibriSpeechMix datasets for both tasks, and delivers acceptable zero-shot performance on multi-talker ASR on AishellMix Mandarin dataset. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_09817 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Empowering Whisper as a Joint Multi-Talker and Target-Talker Speech Recognition System Meng, Lingwei Kang, Jiawen Wang, Yuejiao Jin, Zengrui Wu, Xixin Liu, Xunying Meng, Helen Sound Computation and Language Audio and Speech Processing Multi-talker speech recognition and target-talker speech recognition, both involve transcription in multi-talker contexts, remain significant challenges. However, existing methods rarely attempt to simultaneously address both tasks. In this study, we propose a pioneering approach to empower Whisper, which is a speech foundation model, to tackle joint multi-talker and target-talker speech recognition tasks. Specifically, (i) we freeze Whisper and plug a Sidecar separator into its encoder to separate mixed embedding for multiple talkers; (ii) a Target Talker Identifier is introduced to identify the embedding flow of the target talker on the fly, requiring only three-second enrollment speech as a cue; (iii) soft prompt tuning for decoder is explored for better task adaptation. Our method outperforms previous methods on two- and three-talker LibriMix and LibriSpeechMix datasets for both tasks, and delivers acceptable zero-shot performance on multi-talker ASR on AishellMix Mandarin dataset. |
| title | Empowering Whisper as a Joint Multi-Talker and Target-Talker Speech Recognition System |
| topic | Sound Computation and Language Audio and Speech Processing |
| url | https://arxiv.org/abs/2407.09817 |