Selective Attention Merging for low resource tasks: A case study of Child ASR
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917892914675712 |
|---|---|
| author | Shankar, Natarajan Balaji Wang, Zilai Eren, Eray Alwan, Abeer |
| author_facet | Shankar, Natarajan Balaji Wang, Zilai Eren, Eray Alwan, Abeer |
| contents | While Speech Foundation Models (SFMs) excel in various speech tasks, their performance for low-resource tasks such as child Automatic Speech Recognition (ASR) is hampered by limited pretraining data. To address this, we explore different model merging techniques to leverage knowledge from models trained on larger, more diverse speech corpora. This paper also introduces Selective Attention (SA) Merge, a novel method that selectively merges task vectors from attention matrices to enhance SFM performance on low-resource tasks. Experiments on the MyST database show significant reductions in relative word error rate of up to 14%, outperforming existing model merging and data augmentation techniques. By combining data augmentation techniques with SA Merge, we achieve a new state-of-the-art WER of 8.69 on the MyST database for the Whisper-small model, highlighting the potential of SA Merge for improving low-resource ASR. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_08468 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Selective Attention Merging for low resource tasks: A case study of Child ASR Shankar, Natarajan Balaji Wang, Zilai Eren, Eray Alwan, Abeer Computation and Language Sound Audio and Speech Processing While Speech Foundation Models (SFMs) excel in various speech tasks, their performance for low-resource tasks such as child Automatic Speech Recognition (ASR) is hampered by limited pretraining data. To address this, we explore different model merging techniques to leverage knowledge from models trained on larger, more diverse speech corpora. This paper also introduces Selective Attention (SA) Merge, a novel method that selectively merges task vectors from attention matrices to enhance SFM performance on low-resource tasks. Experiments on the MyST database show significant reductions in relative word error rate of up to 14%, outperforming existing model merging and data augmentation techniques. By combining data augmentation techniques with SA Merge, we achieve a new state-of-the-art WER of 8.69 on the MyST database for the Whisper-small model, highlighting the potential of SA Merge for improving low-resource ASR. |
| title | Selective Attention Merging for low resource tasks: A case study of Child ASR |
| topic | Computation and Language Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2501.08468 |