DiariST: Streaming Speech Translation with Speaker Diarization
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2023
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914648692883456 |
|---|---|
| author | Yang, Mu Kanda, Naoyuki Wang, Xiaofei Chen, Junkun Wang, Peidong Xue, Jian Li, Jinyu Yoshioka, Takuya |
| author_facet | Yang, Mu Kanda, Naoyuki Wang, Xiaofei Chen, Junkun Wang, Peidong Xue, Jian Li, Jinyu Yoshioka, Takuya |
| contents | End-to-end speech translation (ST) for conversation recordings involves several under-explored challenges such as speaker diarization (SD) without accurate word time stamps and handling of overlapping speech in a streaming fashion. In this work, we propose DiariST, the first streaming ST and SD solution. It is built upon a neural transducer-based streaming ST system and integrates token-level serialized output training and t-vector, which were originally developed for multi-talker speech recognition. Due to the absence of evaluation benchmarks in this area, we develop a new evaluation dataset, DiariST-AliMeeting, by translating the reference Chinese transcriptions of the AliMeeting corpus into English. We also propose new metrics, called speaker-agnostic BLEU and speaker-attributed BLEU, to measure the ST quality while taking SD accuracy into account. Our system achieves a strong ST and SD capability compared to offline systems based on Whisper, while performing streaming inference for overlapping speech. To facilitate the research in this new direction, we release the evaluation data, the offline baseline systems, and the evaluation code. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2309_08007 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | DiariST: Streaming Speech Translation with Speaker Diarization Yang, Mu Kanda, Naoyuki Wang, Xiaofei Chen, Junkun Wang, Peidong Xue, Jian Li, Jinyu Yoshioka, Takuya Audio and Speech Processing Computation and Language Sound End-to-end speech translation (ST) for conversation recordings involves several under-explored challenges such as speaker diarization (SD) without accurate word time stamps and handling of overlapping speech in a streaming fashion. In this work, we propose DiariST, the first streaming ST and SD solution. It is built upon a neural transducer-based streaming ST system and integrates token-level serialized output training and t-vector, which were originally developed for multi-talker speech recognition. Due to the absence of evaluation benchmarks in this area, we develop a new evaluation dataset, DiariST-AliMeeting, by translating the reference Chinese transcriptions of the AliMeeting corpus into English. We also propose new metrics, called speaker-agnostic BLEU and speaker-attributed BLEU, to measure the ST quality while taking SD accuracy into account. Our system achieves a strong ST and SD capability compared to offline systems based on Whisper, while performing streaming inference for overlapping speech. To facilitate the research in this new direction, we release the evaluation data, the offline baseline systems, and the evaluation code. |
| title | DiariST: Streaming Speech Translation with Speaker Diarization |
| topic | Audio and Speech Processing Computation and Language Sound |
| url | https://arxiv.org/abs/2309.08007 |