Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2509.09987 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915491763716096 |
|---|---|
| author | Yeh, Sung-Lin Meng, Yen Tang, Hao |
| author_facet | Yeh, Sung-Lin Meng, Yen Tang, Hao |
| contents | There is an increasing interest in obtaining accurate word-level timestamps from strong automatic speech recognizers, in particular Whisper. Existing approaches either require additional training or are simply not competitive. The evaluation in prior work is also relatively loose, typically using a tolerance of more than 200 ms. In this work, we discover attention heads in Whisper that capture accurate word alignments and are distinctively different from those that do not. Moreover, we find that using characters produces finer and more accurate alignments than using wordpieces. Based on these findings, we propose an unsupervised approach to extracting word alignments by filtering attention heads while teacher forcing Whisper with characters. Our approach not only does not require training but also produces word alignments that are more accurate than prior work under a stricter tolerance between 20 ms and 100 ms. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_09987 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Whisper Has an Internal Word Aligner Yeh, Sung-Lin Meng, Yen Tang, Hao Audio and Speech Processing Computation and Language There is an increasing interest in obtaining accurate word-level timestamps from strong automatic speech recognizers, in particular Whisper. Existing approaches either require additional training or are simply not competitive. The evaluation in prior work is also relatively loose, typically using a tolerance of more than 200 ms. In this work, we discover attention heads in Whisper that capture accurate word alignments and are distinctively different from those that do not. Moreover, we find that using characters produces finer and more accurate alignments than using wordpieces. Based on these findings, we propose an unsupervised approach to extracting word alignments by filtering attention heads while teacher forcing Whisper with characters. Our approach not only does not require training but also produces word alignments that are more accurate than prior work under a stricter tolerance between 20 ms and 100 ms. |
| title | Whisper Has an Internal Word Aligner |
| topic | Audio and Speech Processing Computation and Language |
| url | https://arxiv.org/abs/2509.09987 |