SummaryMixing: A Linear-Complexity Alternative to Self-Attention for Speech Recognition and Understanding
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2023
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909249974566912 |
|---|---|
| author | Parcollet, Titouan van Dalen, Rogier Zhang, Shucong Bhattacharya, Sourav |
| author_facet | Parcollet, Titouan van Dalen, Rogier Zhang, Shucong Bhattacharya, Sourav |
| contents | Modern speech processing systems rely on self-attention. Unfortunately, token mixing with self-attention takes quadratic time in the length of the speech utterance, slowing down inference and training and increasing memory consumption. Cheaper alternatives to self-attention for ASR have been developed, but they fail to consistently reach the same level of accuracy. This paper, therefore, proposes a novel linear-time alternative to self-attention. It summarises an utterance with the mean over vectors for all time steps. This single summary is then combined with time-specific information. We call this method "SummaryMixing". Introducing SummaryMixing in state-of-the-art ASR models makes it feasible to preserve or exceed previous speech recognition performance while making training and inference up to 28% faster and reducing memory use by half. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2307_07421 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | SummaryMixing: A Linear-Complexity Alternative to Self-Attention for Speech Recognition and Understanding Parcollet, Titouan van Dalen, Rogier Zhang, Shucong Bhattacharya, Sourav Computation and Language Sound Audio and Speech Processing Modern speech processing systems rely on self-attention. Unfortunately, token mixing with self-attention takes quadratic time in the length of the speech utterance, slowing down inference and training and increasing memory consumption. Cheaper alternatives to self-attention for ASR have been developed, but they fail to consistently reach the same level of accuracy. This paper, therefore, proposes a novel linear-time alternative to self-attention. It summarises an utterance with the mean over vectors for all time steps. This single summary is then combined with time-specific information. We call this method "SummaryMixing". Introducing SummaryMixing in state-of-the-art ASR models makes it feasible to preserve or exceed previous speech recognition performance while making training and inference up to 28% faster and reducing memory use by half. |
| title | SummaryMixing: A Linear-Complexity Alternative to Self-Attention for Speech Recognition and Understanding |
| topic | Computation and Language Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2307.07421 |