Enhancing Whisper's Accuracy and Speed for Indian Languages through Prompt-Tuning and Tokenization
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913627824455680 |
|---|---|
| author | Tripathi, Kumud Gothi, Raj Wasnik, Pankaj |
| author_facet | Tripathi, Kumud Gothi, Raj Wasnik, Pankaj |
| contents | Automatic speech recognition has recently seen a significant advancement with large foundational models such as Whisper. However, these models often struggle to perform well in low-resource languages, such as Indian languages. This paper explores two novel approaches to enhance Whisper's multilingual speech recognition performance in Indian languages. First, we propose prompt-tuning with language family information, which enhances Whisper's accuracy in linguistically similar languages. Second, we introduce a novel tokenizer that reduces the number of generated tokens, thereby accelerating Whisper's inference speed. Our extensive experiments demonstrate that the tokenizer significantly reduces inference time, while prompt-tuning enhances accuracy across various Whisper model sizes, including Small, Medium, and Large. Together, these techniques achieve a balance between optimal WER and inference speed. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_19785 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Enhancing Whisper's Accuracy and Speed for Indian Languages through Prompt-Tuning and Tokenization Tripathi, Kumud Gothi, Raj Wasnik, Pankaj Computation and Language Audio and Speech Processing Automatic speech recognition has recently seen a significant advancement with large foundational models such as Whisper. However, these models often struggle to perform well in low-resource languages, such as Indian languages. This paper explores two novel approaches to enhance Whisper's multilingual speech recognition performance in Indian languages. First, we propose prompt-tuning with language family information, which enhances Whisper's accuracy in linguistically similar languages. Second, we introduce a novel tokenizer that reduces the number of generated tokens, thereby accelerating Whisper's inference speed. Our extensive experiments demonstrate that the tokenizer significantly reduces inference time, while prompt-tuning enhances accuracy across various Whisper model sizes, including Small, Medium, and Large. Together, these techniques achieve a balance between optimal WER and inference speed. |
| title | Enhancing Whisper's Accuracy and Speed for Indian Languages through Prompt-Tuning and Tokenization |
| topic | Computation and Language Audio and Speech Processing |
| url | https://arxiv.org/abs/2412.19785 |