Hypothesis Clustering and Merging: Novel MultiTalker Speech Recognition with Speaker Tokens
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909324021858304 |
|---|---|
| author | Kashiwagi, Yosuke Futami, Hayato Tsunoo, Emiru Arora, Siddhant Watanabe, Shinji |
| author_facet | Kashiwagi, Yosuke Futami, Hayato Tsunoo, Emiru Arora, Siddhant Watanabe, Shinji |
| contents | In many real-world scenarios, such as meetings, multiple speakers are present with an unknown number of participants, and their utterances often overlap. We address these multi-speaker challenges by a novel attention-based encoder-decoder method augmented with special speaker class tokens obtained by speaker clustering. During inference, we select multiple recognition hypotheses conditioned on predicted speaker cluster tokens, and these hypotheses are merged by agglomerative hierarchical clustering (AHC) based on the normalized edit distance. The clustered hypotheses result in the multi-speaker transcriptions with the appropriate number of speakers determined by AHC. Our experiments on the LibriMix dataset demonstrate that our proposed method was particularly effective in complex 3-mix environments, achieving a 55% relative error reduction on clean data and a 36% relative error reduction on noisy data compared with conventional serialized output training. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_15732 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Hypothesis Clustering and Merging: Novel MultiTalker Speech Recognition with Speaker Tokens Kashiwagi, Yosuke Futami, Hayato Tsunoo, Emiru Arora, Siddhant Watanabe, Shinji Computation and Language Sound Audio and Speech Processing In many real-world scenarios, such as meetings, multiple speakers are present with an unknown number of participants, and their utterances often overlap. We address these multi-speaker challenges by a novel attention-based encoder-decoder method augmented with special speaker class tokens obtained by speaker clustering. During inference, we select multiple recognition hypotheses conditioned on predicted speaker cluster tokens, and these hypotheses are merged by agglomerative hierarchical clustering (AHC) based on the normalized edit distance. The clustered hypotheses result in the multi-speaker transcriptions with the appropriate number of speakers determined by AHC. Our experiments on the LibriMix dataset demonstrate that our proposed method was particularly effective in complex 3-mix environments, achieving a 55% relative error reduction on clean data and a 36% relative error reduction on noisy data compared with conventional serialized output training. |
| title | Hypothesis Clustering and Merging: Novel MultiTalker Speech Recognition with Speaker Tokens |
| topic | Computation and Language Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2409.15732 |