M2R-Whisper: Multi-stage and Multi-scale Retrieval Augmentation for Enhancing Whisper
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912270474280960 |
|---|---|
| author | Zhou, Jiaming Zhao, Shiwan He, Jiabei Wang, Hui Zeng, Wenjia Chen, Yong Sun, Haoqin Kong, Aobo Qin, Yong |
| author_facet | Zhou, Jiaming Zhao, Shiwan He, Jiabei Wang, Hui Zeng, Wenjia Chen, Yong Sun, Haoqin Kong, Aobo Qin, Yong |
| contents | State-of-the-art models like OpenAI's Whisper exhibit strong performance in multilingual automatic speech recognition (ASR), but they still face challenges in accurately recognizing diverse subdialects. In this paper, we propose M2R-whisper, a novel multi-stage and multi-scale retrieval augmentation approach designed to enhance ASR performance in low-resource settings. Building on the principles of in-context learning (ICL) and retrieval-augmented techniques, our method employs sentence-level ICL in the pre-processing stage to harness contextual information, while integrating token-level k-Nearest Neighbors (kNN) retrieval as a post-processing step to further refine the final output distribution. By synergistically combining sentence-level and token-level retrieval strategies, M2R-whisper effectively mitigates various types of recognition errors. Experiments conducted on Mandarin and subdialect datasets, including AISHELL-1 and KeSpeech, demonstrate substantial improvements in ASR accuracy, all achieved without any parameter updates. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_11889 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | M2R-Whisper: Multi-stage and Multi-scale Retrieval Augmentation for Enhancing Whisper Zhou, Jiaming Zhao, Shiwan He, Jiabei Wang, Hui Zeng, Wenjia Chen, Yong Sun, Haoqin Kong, Aobo Qin, Yong Sound Audio and Speech Processing State-of-the-art models like OpenAI's Whisper exhibit strong performance in multilingual automatic speech recognition (ASR), but they still face challenges in accurately recognizing diverse subdialects. In this paper, we propose M2R-whisper, a novel multi-stage and multi-scale retrieval augmentation approach designed to enhance ASR performance in low-resource settings. Building on the principles of in-context learning (ICL) and retrieval-augmented techniques, our method employs sentence-level ICL in the pre-processing stage to harness contextual information, while integrating token-level k-Nearest Neighbors (kNN) retrieval as a post-processing step to further refine the final output distribution. By synergistically combining sentence-level and token-level retrieval strategies, M2R-whisper effectively mitigates various types of recognition errors. Experiments conducted on Mandarin and subdialect datasets, including AISHELL-1 and KeSpeech, demonstrate substantial improvements in ASR accuracy, all achieved without any parameter updates. |
| title | M2R-Whisper: Multi-stage and Multi-scale Retrieval Augmentation for Enhancing Whisper |
| topic | Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2409.11889 |