Hard-Synth: Synthesizing Diverse Hard Samples for ASR using Zero-Shot TTS and LLM
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913581846495232 |
|---|---|
| author | Yu, Jiawei Li, Yuang Qiao, Xiaosong Zhao, Huan Zhao, Xiaofeng Tang, Wei Zhang, Min Yang, Hao Su, Jinsong |
| author_facet | Yu, Jiawei Li, Yuang Qiao, Xiaosong Zhao, Huan Zhao, Xiaofeng Tang, Wei Zhang, Min Yang, Hao Su, Jinsong |
| contents | Text-to-speech (TTS) models have been widely adopted to enhance automatic speech recognition (ASR) systems using text-only corpora, thereby reducing the cost of labeling real speech data. Existing research primarily utilizes additional text data and predefined speech styles supported by TTS models. In this paper, we propose Hard-Synth, a novel ASR data augmentation method that leverages large language models (LLMs) and advanced zero-shot TTS. Our approach employs LLMs to generate diverse in-domain text through rewriting, without relying on additional text data. Rather than using predefined speech styles, we introduce a hard prompt selection method with zero-shot TTS to clone speech styles that the ASR model finds challenging to recognize. Experiments demonstrate that Hard-Synth significantly enhances the Conformer model, achieving relative word error rate (WER) reductions of 6.5\%/4.4\% on LibriSpeech dev/test-other subsets. Additionally, we show that Hard-Synth is data-efficient and capable of reducing bias in ASR. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_13159 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Hard-Synth: Synthesizing Diverse Hard Samples for ASR using Zero-Shot TTS and LLM Yu, Jiawei Li, Yuang Qiao, Xiaosong Zhao, Huan Zhao, Xiaofeng Tang, Wei Zhang, Min Yang, Hao Su, Jinsong Computation and Language Sound Audio and Speech Processing Text-to-speech (TTS) models have been widely adopted to enhance automatic speech recognition (ASR) systems using text-only corpora, thereby reducing the cost of labeling real speech data. Existing research primarily utilizes additional text data and predefined speech styles supported by TTS models. In this paper, we propose Hard-Synth, a novel ASR data augmentation method that leverages large language models (LLMs) and advanced zero-shot TTS. Our approach employs LLMs to generate diverse in-domain text through rewriting, without relying on additional text data. Rather than using predefined speech styles, we introduce a hard prompt selection method with zero-shot TTS to clone speech styles that the ASR model finds challenging to recognize. Experiments demonstrate that Hard-Synth significantly enhances the Conformer model, achieving relative word error rate (WER) reductions of 6.5\%/4.4\% on LibriSpeech dev/test-other subsets. Additionally, we show that Hard-Synth is data-efficient and capable of reducing bias in ASR. |
| title | Hard-Synth: Synthesizing Diverse Hard Samples for ASR using Zero-Shot TTS and LLM |
| topic | Computation and Language Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2411.13159 |