A Self-Refining Framework for Enhancing ASR Using TTS-Synthesized Data
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908408525881344 |
|---|---|
| author | Chou, Cheng-Kang Hsu, Chan-Jan Chung, Ho-Lam Tseng, Liang-Hsuan Cheng, Hsi-Chun Fu, Yu-Kuan Huang, Kuan Po Lee, Hung-Yi |
| author_facet | Chou, Cheng-Kang Hsu, Chan-Jan Chung, Ho-Lam Tseng, Liang-Hsuan Cheng, Hsi-Chun Fu, Yu-Kuan Huang, Kuan Po Lee, Hung-Yi |
| contents | We propose a self-refining framework that enhances ASR performance with only unlabeled datasets. The process starts with an existing ASR model generating pseudo-labels on unannotated speech, which are then used to train a high-fidelity text-to-speech (TTS) system. Then, synthesized speech text pairs are bootstrapped into the original ASR system, completing the closed-loop self-improvement cycle. We demonstrated the effectiveness of the framework on Taiwanese Mandarin speech. Leveraging 6,000 hours of unlabeled speech, a moderate amount of text data, and synthetic content from the AI models, we adapt Whisper-large-v2 into a specialized model, Twister. Twister reduces error rates by up to 20% on Mandarin and 50% on Mandarin-English code-switching benchmarks compared to Whisper. Results highlight the framework as a compelling alternative to pseudo-labeling self-distillation approaches and provides a practical pathway for improving ASR performance in low-resource or domain-specific settings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_11130 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | A Self-Refining Framework for Enhancing ASR Using TTS-Synthesized Data Chou, Cheng-Kang Hsu, Chan-Jan Chung, Ho-Lam Tseng, Liang-Hsuan Cheng, Hsi-Chun Fu, Yu-Kuan Huang, Kuan Po Lee, Hung-Yi Computation and Language Artificial Intelligence Sound Audio and Speech Processing We propose a self-refining framework that enhances ASR performance with only unlabeled datasets. The process starts with an existing ASR model generating pseudo-labels on unannotated speech, which are then used to train a high-fidelity text-to-speech (TTS) system. Then, synthesized speech text pairs are bootstrapped into the original ASR system, completing the closed-loop self-improvement cycle. We demonstrated the effectiveness of the framework on Taiwanese Mandarin speech. Leveraging 6,000 hours of unlabeled speech, a moderate amount of text data, and synthetic content from the AI models, we adapt Whisper-large-v2 into a specialized model, Twister. Twister reduces error rates by up to 20% on Mandarin and 50% on Mandarin-English code-switching benchmarks compared to Whisper. Results highlight the framework as a compelling alternative to pseudo-labeling self-distillation approaches and provides a practical pathway for improving ASR performance in low-resource or domain-specific settings. |
| title | A Self-Refining Framework for Enhancing ASR Using TTS-Synthesized Data |
| topic | Computation and Language Artificial Intelligence Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2506.11130 |