Investigation of Whisper ASR Hallucinations Induced by Non-Speech Audio
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916653215776768 |
|---|---|
| author | Barański, Mateusz Jasiński, Jan Bartolewska, Julitta Kacprzak, Stanisław Witkowski, Marcin Kowalczyk, Konrad |
| author_facet | Barański, Mateusz Jasiński, Jan Bartolewska, Julitta Kacprzak, Stanisław Witkowski, Marcin Kowalczyk, Konrad |
| contents | Hallucinations of deep neural models are amongst key challenges in automatic speech recognition (ASR). In this paper, we investigate hallucinations of the Whisper ASR model induced by non-speech audio segments present during inference. By inducting hallucinations with various types of sounds, we show that there exists a set of hallucinations that appear frequently. We then study hallucinations caused by the augmentation of speech with such sounds. Finally, we describe the creation of a bag of hallucinations (BoH) that allows to remove the effect of hallucinations through the post-processing of text transcriptions. The results of our experiments show that such post-processing is capable of reducing word error rate (WER) and acts as a good safeguard against problematic hallucinations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_11378 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Investigation of Whisper ASR Hallucinations Induced by Non-Speech Audio Barański, Mateusz Jasiński, Jan Bartolewska, Julitta Kacprzak, Stanisław Witkowski, Marcin Kowalczyk, Konrad Sound Artificial Intelligence Audio and Speech Processing Hallucinations of deep neural models are amongst key challenges in automatic speech recognition (ASR). In this paper, we investigate hallucinations of the Whisper ASR model induced by non-speech audio segments present during inference. By inducting hallucinations with various types of sounds, we show that there exists a set of hallucinations that appear frequently. We then study hallucinations caused by the augmentation of speech with such sounds. Finally, we describe the creation of a bag of hallucinations (BoH) that allows to remove the effect of hallucinations through the post-processing of text transcriptions. The results of our experiments show that such post-processing is capable of reducing word error rate (WER) and acts as a good safeguard against problematic hallucinations. |
| title | Investigation of Whisper ASR Hallucinations Induced by Non-Speech Audio |
| topic | Sound Artificial Intelligence Audio and Speech Processing |
| url | https://arxiv.org/abs/2501.11378 |