Folding Attention: Memory and Power Optimization for On-Device Transformer-based Streaming Speech Recognition
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2023
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866929215248531456 |
|---|---|
| author | Li, Yang Lai, Liangzhen Shangguan, Yuan Iandola, Forrest N. Ni, Zhaoheng Chang, Ernie Shi, Yangyang Chandra, Vikas |
| author_facet | Li, Yang Lai, Liangzhen Shangguan, Yuan Iandola, Forrest N. Ni, Zhaoheng Chang, Ernie Shi, Yangyang Chandra, Vikas |
| contents | Transformer-based models excel in speech recognition. Existing efforts to optimize Transformer inference, typically for long-context applications, center on simplifying attention score calculations. However, streaming speech recognition models usually process a limited number of tokens each time, making attention score calculation less of a bottleneck. Instead, the bottleneck lies in the linear projection layers of multi-head attention and feedforward networks, constituting a substantial portion of the model size and contributing significantly to computation, memory, and power usage.
To address this bottleneck, we propose folding attention, a technique targeting these linear layers, significantly reducing model size and improving memory and power efficiency. Experiments on on-device Transformer-based streaming speech recognition models show that folding attention reduces model size (and corresponding memory consumption) by up to 24% and power consumption by up to 23%, all without compromising model accuracy or computation overhead. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2309_07988 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Folding Attention: Memory and Power Optimization for On-Device Transformer-based Streaming Speech Recognition Li, Yang Lai, Liangzhen Shangguan, Yuan Iandola, Forrest N. Ni, Zhaoheng Chang, Ernie Shi, Yangyang Chandra, Vikas Machine Learning Hardware Architecture Sound Audio and Speech Processing Transformer-based models excel in speech recognition. Existing efforts to optimize Transformer inference, typically for long-context applications, center on simplifying attention score calculations. However, streaming speech recognition models usually process a limited number of tokens each time, making attention score calculation less of a bottleneck. Instead, the bottleneck lies in the linear projection layers of multi-head attention and feedforward networks, constituting a substantial portion of the model size and contributing significantly to computation, memory, and power usage. To address this bottleneck, we propose folding attention, a technique targeting these linear layers, significantly reducing model size and improving memory and power efficiency. Experiments on on-device Transformer-based streaming speech recognition models show that folding attention reduces model size (and corresponding memory consumption) by up to 24% and power consumption by up to 23%, all without compromising model accuracy or computation overhead. |
| title | Folding Attention: Memory and Power Optimization for On-Device Transformer-based Streaming Speech Recognition |
| topic | Machine Learning Hardware Architecture Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2309.07988 |