Multi-Stage Multi-Modal Pre-Training for Automatic Speech Recognition
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913289571663872 |
|---|---|
| author | Jain, Yash Chan, David Dheram, Pranav Khare, Aparna Shonibare, Olabanji Ravichandran, Venkatesh Ghosh, Shalini |
| author_facet | Jain, Yash Chan, David Dheram, Pranav Khare, Aparna Shonibare, Olabanji Ravichandran, Venkatesh Ghosh, Shalini |
| contents | Recent advances in machine learning have demonstrated that multi-modal pre-training can improve automatic speech recognition (ASR) performance compared to randomly initialized models, even when models are fine-tuned on uni-modal tasks. Existing multi-modal pre-training methods for the ASR task have primarily focused on single-stage pre-training where a single unsupervised task is used for pre-training followed by fine-tuning on the downstream task. In this work, we introduce a novel method combining multi-modal and multi-task unsupervised pre-training with a translation-based supervised mid-training approach. We empirically demonstrate that such a multi-stage approach leads to relative word error rate (WER) improvements of up to 38.45% over baselines on both Librispeech and SUPERB. Additionally, we share several important findings for choosing pre-training methods and datasets. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_19822 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Multi-Stage Multi-Modal Pre-Training for Automatic Speech Recognition Jain, Yash Chan, David Dheram, Pranav Khare, Aparna Shonibare, Olabanji Ravichandran, Venkatesh Ghosh, Shalini Computation and Language Artificial Intelligence Recent advances in machine learning have demonstrated that multi-modal pre-training can improve automatic speech recognition (ASR) performance compared to randomly initialized models, even when models are fine-tuned on uni-modal tasks. Existing multi-modal pre-training methods for the ASR task have primarily focused on single-stage pre-training where a single unsupervised task is used for pre-training followed by fine-tuning on the downstream task. In this work, we introduce a novel method combining multi-modal and multi-task unsupervised pre-training with a translation-based supervised mid-training approach. We empirically demonstrate that such a multi-stage approach leads to relative word error rate (WER) improvements of up to 38.45% over baselines on both Librispeech and SUPERB. Additionally, we share several important findings for choosing pre-training methods and datasets. |
| title | Multi-Stage Multi-Modal Pre-Training for Automatic Speech Recognition |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2403.19822 |