SoftmAP: Software-Hardware Co-design for Integer-Only Softmax on Associative Processors
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916497027235840 |
|---|---|
| author | Rakka, Mariam Li, Jinhao Dai, Guohao Eltawil, Ahmed Fouda, Mohammed E. Kurdahi, Fadi |
| author_facet | Rakka, Mariam Li, Jinhao Dai, Guohao Eltawil, Ahmed Fouda, Mohammed E. Kurdahi, Fadi |
| contents | Recent research efforts focus on reducing the computational and memory overheads of Large Language Models (LLMs) to make them feasible on resource-constrained devices. Despite advancements in compression techniques, non-linear operators like Softmax and Layernorm remain bottlenecks due to their sensitivity to quantization. We propose SoftmAP, a software-hardware co-design methodology that implements an integer-only low-precision Softmax using In-Memory Compute (IMC) hardware. Our method achieves up to three orders of magnitude improvement in the energy-delay product compared to A100 and RTX3090 GPUs, making LLMs more deployable without compromising performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_17847 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SoftmAP: Software-Hardware Co-design for Integer-Only Softmax on Associative Processors Rakka, Mariam Li, Jinhao Dai, Guohao Eltawil, Ahmed Fouda, Mohammed E. Kurdahi, Fadi Hardware Architecture Artificial Intelligence Recent research efforts focus on reducing the computational and memory overheads of Large Language Models (LLMs) to make them feasible on resource-constrained devices. Despite advancements in compression techniques, non-linear operators like Softmax and Layernorm remain bottlenecks due to their sensitivity to quantization. We propose SoftmAP, a software-hardware co-design methodology that implements an integer-only low-precision Softmax using In-Memory Compute (IMC) hardware. Our method achieves up to three orders of magnitude improvement in the energy-delay product compared to A100 and RTX3090 GPUs, making LLMs more deployable without compromising performance. |
| title | SoftmAP: Software-Hardware Co-design for Integer-Only Softmax on Associative Processors |
| topic | Hardware Architecture Artificial Intelligence |
| url | https://arxiv.org/abs/2411.17847 |