Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2023
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2312.14378 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916119078502400 |
|---|---|
| author | Sundar, Anirudh S. Yang, Chao-Han Huck Chan, David M. Ghosh, Shalini Ravichandran, Venkatesh Nidadavolu, Phani Sankar |
| author_facet | Sundar, Anirudh S. Yang, Chao-Han Huck Chan, David M. Ghosh, Shalini Ravichandran, Venkatesh Nidadavolu, Phani Sankar |
| contents | Training large foundation models using self-supervised objectives on unlabeled data, followed by fine-tuning on downstream tasks, has emerged as a standard procedure. Unfortunately, the efficacy of this approach is often constrained by both limited fine-tuning compute and scarcity in labeled downstream data. We introduce Multimodal Attention Merging (MAM), an attempt that facilitates direct knowledge transfer from attention matrices of models rooted in high resource modalities, text and images, to those in resource-constrained domains, speech and audio, employing a zero-shot paradigm. MAM reduces the relative Word Error Rate (WER) of an Automatic Speech Recognition (ASR) model by up to 6.70%, and relative classification error of an Audio Event Classification (AEC) model by 10.63%. In cases where some data/compute is available, we present Learnable-MAM, a data-driven approach to merging attention matrices, resulting in a further 2.90% relative reduction in WER for ASR and 18.42% relative reduction in AEC compared to fine-tuning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_14378 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Multimodal Attention Merging for Improved Speech Recognition and Audio Event Classification Sundar, Anirudh S. Yang, Chao-Han Huck Chan, David M. Ghosh, Shalini Ravichandran, Venkatesh Nidadavolu, Phani Sankar Machine Learning Sound Audio and Speech Processing Training large foundation models using self-supervised objectives on unlabeled data, followed by fine-tuning on downstream tasks, has emerged as a standard procedure. Unfortunately, the efficacy of this approach is often constrained by both limited fine-tuning compute and scarcity in labeled downstream data. We introduce Multimodal Attention Merging (MAM), an attempt that facilitates direct knowledge transfer from attention matrices of models rooted in high resource modalities, text and images, to those in resource-constrained domains, speech and audio, employing a zero-shot paradigm. MAM reduces the relative Word Error Rate (WER) of an Automatic Speech Recognition (ASR) model by up to 6.70%, and relative classification error of an Audio Event Classification (AEC) model by 10.63%. In cases where some data/compute is available, we present Learnable-MAM, a data-driven approach to merging attention matrices, resulting in a further 2.90% relative reduction in WER for ASR and 18.42% relative reduction in AEC compared to fine-tuning. |
| title | Multimodal Attention Merging for Improved Speech Recognition and Audio Event Classification |
| topic | Machine Learning Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2312.14378 |