Hierarchical Recurrent Adapters for Efficient Multi-Task Adaptation of Large Speech Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909154587705344 |
|---|---|
| author | Munkhdalai, Tsendsuren Chen, Youzheng Sim, Khe Chai Biadsy, Fadi Sainath, Tara Mengibar, Pedro Moreno |
| author_facet | Munkhdalai, Tsendsuren Chen, Youzheng Sim, Khe Chai Biadsy, Fadi Sainath, Tara Mengibar, Pedro Moreno |
| contents | Parameter efficient adaptation methods have become a key mechanism to train large pre-trained models for downstream tasks. However, their per-task parameter overhead is considered still high when the number of downstream tasks to adapt for is large. We introduce an adapter module that has a better efficiency in large scale multi-task adaptation scenario. Our adapter is hierarchical in terms of how the adapter parameters are allocated. The adapter consists of a single shared controller network and multiple task-level adapter heads to reduce the per-task parameter overhead without performance regression on downstream tasks. The adapter is also recurrent so the entire adapter parameters are reused across different layers of the pre-trained model. Our Hierarchical Recurrent Adapter (HRA) outperforms the previous adapter-based approaches as well as full model fine-tuning baseline in both single and multi-task adaptation settings when evaluated on automatic speech recognition tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_19709 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Hierarchical Recurrent Adapters for Efficient Multi-Task Adaptation of Large Speech Models Munkhdalai, Tsendsuren Chen, Youzheng Sim, Khe Chai Biadsy, Fadi Sainath, Tara Mengibar, Pedro Moreno Audio and Speech Processing Artificial Intelligence Computation and Language Machine Learning Neural and Evolutionary Computing Parameter efficient adaptation methods have become a key mechanism to train large pre-trained models for downstream tasks. However, their per-task parameter overhead is considered still high when the number of downstream tasks to adapt for is large. We introduce an adapter module that has a better efficiency in large scale multi-task adaptation scenario. Our adapter is hierarchical in terms of how the adapter parameters are allocated. The adapter consists of a single shared controller network and multiple task-level adapter heads to reduce the per-task parameter overhead without performance regression on downstream tasks. The adapter is also recurrent so the entire adapter parameters are reused across different layers of the pre-trained model. Our Hierarchical Recurrent Adapter (HRA) outperforms the previous adapter-based approaches as well as full model fine-tuning baseline in both single and multi-task adaptation settings when evaluated on automatic speech recognition tasks. |
| title | Hierarchical Recurrent Adapters for Efficient Multi-Task Adaptation of Large Speech Models |
| topic | Audio and Speech Processing Artificial Intelligence Computation and Language Machine Learning Neural and Evolutionary Computing |
| url | https://arxiv.org/abs/2403.19709 |