Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2505.23627 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908384305872896 |
|---|---|
| author | Smith, Griffin Dietz Yee, Dianna Chen, Jennifer King Findlater, Leah |
| author_facet | Smith, Griffin Dietz Yee, Dianna Chen, Jennifer King Findlater, Leah |
| contents | Identifying mistakes (i.e., miscues) made while reading aloud is commonly approached post-hoc by comparing automatic speech recognition (ASR) transcriptions to the target reading text. However, post-hoc methods perform poorly when ASR inaccurately transcribes verbatim speech. To improve on current methods for reading error annotation, we propose a novel end-to-end architecture that incorporates the target reading text via prompting and is trained for both improved verbatim transcription and direct miscue detection. Our contributions include: first, demonstrating that incorporating reading text through prompting benefits verbatim transcription performance over fine-tuning, and second, showing that it is feasible to augment speech recognition tasks for end-to-end miscue detection. We conducted two case studies -- children's read-aloud and adult atypical speech -- and found that our proposed strategies improve verbatim transcription and miscue detection compared to current state-of-the-art. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_23627 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Prompting Whisper for Improved Verbatim Transcription and End-to-end Miscue Detection Smith, Griffin Dietz Yee, Dianna Chen, Jennifer King Findlater, Leah Machine Learning Identifying mistakes (i.e., miscues) made while reading aloud is commonly approached post-hoc by comparing automatic speech recognition (ASR) transcriptions to the target reading text. However, post-hoc methods perform poorly when ASR inaccurately transcribes verbatim speech. To improve on current methods for reading error annotation, we propose a novel end-to-end architecture that incorporates the target reading text via prompting and is trained for both improved verbatim transcription and direct miscue detection. Our contributions include: first, demonstrating that incorporating reading text through prompting benefits verbatim transcription performance over fine-tuning, and second, showing that it is feasible to augment speech recognition tasks for end-to-end miscue detection. We conducted two case studies -- children's read-aloud and adult atypical speech -- and found that our proposed strategies improve verbatim transcription and miscue detection compared to current state-of-the-art. |
| title | Prompting Whisper for Improved Verbatim Transcription and End-to-end Miscue Detection |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2505.23627 |