Simul-Whisper: Attention-Guided Streaming Whisper with Truncation Detection
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913817808601088 |
|---|---|
| author | Wang, Haoyu Hu, Guoqiang Lin, Guodong Zhang, Wei-Qiang Li, Jian |
| author_facet | Wang, Haoyu Hu, Guoqiang Lin, Guodong Zhang, Wei-Qiang Li, Jian |
| contents | As a robust and large-scale multilingual speech recognition model, Whisper has demonstrated impressive results in many low-resource and out-of-distribution scenarios. However, its encoder-decoder structure hinders its application to streaming speech recognition. In this paper, we introduce Simul-Whisper, which uses the time alignment embedded in Whisper's cross-attention to guide auto-regressive decoding and achieve chunk-based streaming ASR without any fine-tuning of the pre-trained model. Furthermore, we observe the negative effect of the truncated words at the chunk boundaries on the decoding results and propose an integrate-and-fire-based truncation detection model to address this issue. Experiments on multiple languages and Whisper architectures show that Simul-Whisper achieves an average absolute word error rate degradation of only 1.46% at a chunk size of 1 second, which significantly outperforms the current state-of-the-art baseline. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_10052 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Simul-Whisper: Attention-Guided Streaming Whisper with Truncation Detection Wang, Haoyu Hu, Guoqiang Lin, Guodong Zhang, Wei-Qiang Li, Jian Sound Computation and Language Audio and Speech Processing As a robust and large-scale multilingual speech recognition model, Whisper has demonstrated impressive results in many low-resource and out-of-distribution scenarios. However, its encoder-decoder structure hinders its application to streaming speech recognition. In this paper, we introduce Simul-Whisper, which uses the time alignment embedded in Whisper's cross-attention to guide auto-regressive decoding and achieve chunk-based streaming ASR without any fine-tuning of the pre-trained model. Furthermore, we observe the negative effect of the truncated words at the chunk boundaries on the decoding results and propose an integrate-and-fire-based truncation detection model to address this issue. Experiments on multiple languages and Whisper architectures show that Simul-Whisper achieves an average absolute word error rate degradation of only 1.46% at a chunk size of 1 second, which significantly outperforms the current state-of-the-art baseline. |
| title | Simul-Whisper: Attention-Guided Streaming Whisper with Truncation Detection |
| topic | Sound Computation and Language Audio and Speech Processing |
| url | https://arxiv.org/abs/2406.10052 |