Analog In-Memory Computing Attention Mechanism for Fast and Energy-Efficient Large Language Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866912131509649408 |
|---|---|
| author | Leroux, Nathan Manea, Paul-Philipp Sudarshan, Chirag Finkbeiner, Jan Siegel, Sebastian Strachan, John Paul Neftci, Emre |
| author_facet | Leroux, Nathan Manea, Paul-Philipp Sudarshan, Chirag Finkbeiner, Jan Siegel, Sebastian Strachan, John Paul Neftci, Emre |
| contents | Transformer networks, driven by self-attention, are central to Large Language Models. In generative Transformers, self-attention uses cache memory to store token projections, avoiding recomputation at each time step. However, GPU-stored projections must be loaded into SRAM for each new generation step, causing latency and energy bottlenecks.
We present a custom self-attention in-memory computing architecture based on emerging charge-based memories called gain cells, which can be efficiently written to store new tokens during sequence generation and enable parallel analog dot-product computation required for self-attention. However, the analog gain cell circuits introduce non-idealities and constraints preventing the direct mapping of pre-trained models. To circumvent this problem, we design an initialization algorithm achieving text processing performance comparable to GPT-2 without training from scratch. Our architecture respectively reduces attention latency and energy consumption by up to two and five orders of magnitude compared to GPUs, marking a significant step toward ultra-fast, low-power generative Transformers. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_19315 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Analog In-Memory Computing Attention Mechanism for Fast and Energy-Efficient Large Language Models Leroux, Nathan Manea, Paul-Philipp Sudarshan, Chirag Finkbeiner, Jan Siegel, Sebastian Strachan, John Paul Neftci, Emre Neural and Evolutionary Computing Artificial Intelligence Hardware Architecture Emerging Technologies Transformer networks, driven by self-attention, are central to Large Language Models. In generative Transformers, self-attention uses cache memory to store token projections, avoiding recomputation at each time step. However, GPU-stored projections must be loaded into SRAM for each new generation step, causing latency and energy bottlenecks. We present a custom self-attention in-memory computing architecture based on emerging charge-based memories called gain cells, which can be efficiently written to store new tokens during sequence generation and enable parallel analog dot-product computation required for self-attention. However, the analog gain cell circuits introduce non-idealities and constraints preventing the direct mapping of pre-trained models. To circumvent this problem, we design an initialization algorithm achieving text processing performance comparable to GPT-2 without training from scratch. Our architecture respectively reduces attention latency and energy consumption by up to two and five orders of magnitude compared to GPUs, marking a significant step toward ultra-fast, low-power generative Transformers. |
| title | Analog In-Memory Computing Attention Mechanism for Fast and Energy-Efficient Large Language Models |
| topic | Neural and Evolutionary Computing Artificial Intelligence Hardware Architecture Emerging Technologies |
| url | https://arxiv.org/abs/2409.19315 |