An Analysis of Linear Complexity Attention Substitutes with BEST-RQ
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912014522122240 |
|---|---|
| author | Whetten, Ryan Parcollet, Titouan Moumen, Adel Dinarelli, Marco Estève, Yannick |
| author_facet | Whetten, Ryan Parcollet, Titouan Moumen, Adel Dinarelli, Marco Estève, Yannick |
| contents | Self-Supervised Learning (SSL) has proven to be effective in various domains, including speech processing. However, SSL is computationally and memory expensive. This is in part due the quadratic complexity of multi-head self-attention (MHSA). Alternatives for MHSA have been proposed and used in the speech domain, but have yet to be investigated properly in an SSL setting. In this work, we study the effects of replacing MHSA with recent state-of-the-art alternatives that have linear complexity, namely, HyperMixing, Fastformer, SummaryMixing, and Mamba. We evaluate these methods by looking at the speed, the amount of VRAM consumed, and the performance on the SSL MP3S benchmark. Results show that these linear alternatives maintain competitive performance compared to MHSA while, on average, decreasing VRAM consumption by around 20% to 60% and increasing speed from 7% to 65% for input sequences ranging from 20 to 80 seconds. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_02596 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | An Analysis of Linear Complexity Attention Substitutes with BEST-RQ Whetten, Ryan Parcollet, Titouan Moumen, Adel Dinarelli, Marco Estève, Yannick Machine Learning Computation and Language Sound Audio and Speech Processing Self-Supervised Learning (SSL) has proven to be effective in various domains, including speech processing. However, SSL is computationally and memory expensive. This is in part due the quadratic complexity of multi-head self-attention (MHSA). Alternatives for MHSA have been proposed and used in the speech domain, but have yet to be investigated properly in an SSL setting. In this work, we study the effects of replacing MHSA with recent state-of-the-art alternatives that have linear complexity, namely, HyperMixing, Fastformer, SummaryMixing, and Mamba. We evaluate these methods by looking at the speed, the amount of VRAM consumed, and the performance on the SSL MP3S benchmark. Results show that these linear alternatives maintain competitive performance compared to MHSA while, on average, decreasing VRAM consumption by around 20% to 60% and increasing speed from 7% to 65% for input sequences ranging from 20 to 80 seconds. |
| title | An Analysis of Linear Complexity Attention Substitutes with BEST-RQ |
| topic | Machine Learning Computation and Language Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2409.02596 |