Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2408.16589 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914928902799360 |
|---|---|
| author | Wagner, Laurin Thallinger, Bernhard Zusag, Mario |
| author_facet | Wagner, Laurin Thallinger, Bernhard Zusag, Mario |
| contents | We demonstrate that carefully adjusting the tokenizer of the Whisper speech recognition model significantly improves the precision of word-level timestamps when applying dynamic time warping to the decoder's cross-attention scores. We fine-tune the model to produce more verbatim speech transcriptions and employ several techniques to increase robustness against multiple speakers and background noise. These adjustments achieve state-of-the-art performance on benchmarks for verbatim speech transcription, word segmentation, and the timed detection of filler events, and can further mitigate transcription hallucinations. The code is available open https://github.com/nyrahealth/CrisperWhisper. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_16589 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | CrisperWhisper: Accurate Timestamps on Verbatim Speech Transcriptions Wagner, Laurin Thallinger, Bernhard Zusag, Mario Machine Learning We demonstrate that carefully adjusting the tokenizer of the Whisper speech recognition model significantly improves the precision of word-level timestamps when applying dynamic time warping to the decoder's cross-attention scores. We fine-tune the model to produce more verbatim speech transcriptions and employ several techniques to increase robustness against multiple speakers and background noise. These adjustments achieve state-of-the-art performance on benchmarks for verbatim speech transcription, word segmentation, and the timed detection of filler events, and can further mitigate transcription hallucinations. The code is available open https://github.com/nyrahealth/CrisperWhisper. |
| title | CrisperWhisper: Accurate Timestamps on Verbatim Speech Transcriptions |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2408.16589 |