Full-Rank No More: Low-Rank Weight Training for Modern Speech Recognition Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909344280346624 |
|---|---|
| author | Fernandez-Lopez, Adriana Liu, Shiwei Yin, Lu Petridis, Stavros Pantic, Maja |
| author_facet | Fernandez-Lopez, Adriana Liu, Shiwei Yin, Lu Petridis, Stavros Pantic, Maja |
| contents | This paper investigates the under-explored area of low-rank weight training for large-scale Conformer-based speech recognition models from scratch. Our study demonstrates the viability of this training paradigm for such models, yielding several notable findings. Firstly, we discover that applying a low-rank structure exclusively to the attention modules can unexpectedly enhance performance, even with a significant rank reduction of 12%. In contrast, feed-forward layers present greater challenges, as they begin to exhibit performance degradation with a moderate 50% rank reduction. Furthermore, we find that both initialization and layer-wise rank assignment play critical roles in successful low-rank training. Specifically, employing SVD initialization and linear layer-wise rank mapping significantly boosts the efficacy of low-rank weight training. Building on these insights, we introduce the Low-Rank Speech Model from Scratch (LR-SMS), an approach that achieves performance parity with full-rank training while delivering substantial reductions in parameters count (by at least 2x), and training time speedups (by 1.3x for ASR and 1.15x for AVSR). |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_07771 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Full-Rank No More: Low-Rank Weight Training for Modern Speech Recognition Models Fernandez-Lopez, Adriana Liu, Shiwei Yin, Lu Petridis, Stavros Pantic, Maja Sound Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition Audio and Speech Processing This paper investigates the under-explored area of low-rank weight training for large-scale Conformer-based speech recognition models from scratch. Our study demonstrates the viability of this training paradigm for such models, yielding several notable findings. Firstly, we discover that applying a low-rank structure exclusively to the attention modules can unexpectedly enhance performance, even with a significant rank reduction of 12%. In contrast, feed-forward layers present greater challenges, as they begin to exhibit performance degradation with a moderate 50% rank reduction. Furthermore, we find that both initialization and layer-wise rank assignment play critical roles in successful low-rank training. Specifically, employing SVD initialization and linear layer-wise rank mapping significantly boosts the efficacy of low-rank weight training. Building on these insights, we introduce the Low-Rank Speech Model from Scratch (LR-SMS), an approach that achieves performance parity with full-rank training while delivering substantial reductions in parameters count (by at least 2x), and training time speedups (by 1.3x for ASR and 1.15x for AVSR). |
| title | Full-Rank No More: Low-Rank Weight Training for Modern Speech Recognition Models |
| topic | Sound Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition Audio and Speech Processing |
| url | https://arxiv.org/abs/2410.07771 |