LPU: A Latency-Optimized and Highly Scalable Processor for Large Language Model Inference
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866914912232538112 |
|---|---|
| author | Moon, Seungjae Kim, Jung-Hoon Kim, Junsoo Hong, Seongmin Cha, Junseo Kim, Minsu Lim, Sukbin Choi, Gyubin Seo, Dongjin Kim, Jongho Lee, Hunjong Park, Hyunjun Ko, Ryeowook Choi, Soongyu Park, Jongse Lee, Jinwon Kim, Joo-Young |
| author_facet | Moon, Seungjae Kim, Jung-Hoon Kim, Junsoo Hong, Seongmin Cha, Junseo Kim, Minsu Lim, Sukbin Choi, Gyubin Seo, Dongjin Kim, Jongho Lee, Hunjong Park, Hyunjun Ko, Ryeowook Choi, Soongyu Park, Jongse Lee, Jinwon Kim, Joo-Young |
| contents | The explosive arrival of OpenAI's ChatGPT has fueled the globalization of large language model (LLM), which consists of billions of pretrained parameters that embodies the aspects of syntax and semantics. HyperAccel introduces latency processing unit (LPU), a latency-optimized and highly scalable processor architecture for the acceleration of LLM inference. LPU perfectly balances the memory bandwidth and compute logic with streamlined dataflow to maximize performance and efficiency. LPU is equipped with expandable synchronization link (ESL) that hides data synchronization latency between multiple LPUs. HyperDex complements LPU as an intuitive software framework to run LLM applications. LPU achieves 1.25 ms/token and 20.9 ms/token for 1.3B and 66B model, respectively, which is 2.09x and 1.37x faster than the GPU. LPU, synthesized using Samsung 4nm process, has total area of 0.824 mm2 and power consumption of 284.31 mW. LPU-based servers achieve 1.33x and 1.32x energy efficiency over NVIDIA H100 and L4 servers, respectively. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_07326 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LPU: A Latency-Optimized and Highly Scalable Processor for Large Language Model Inference Moon, Seungjae Kim, Jung-Hoon Kim, Junsoo Hong, Seongmin Cha, Junseo Kim, Minsu Lim, Sukbin Choi, Gyubin Seo, Dongjin Kim, Jongho Lee, Hunjong Park, Hyunjun Ko, Ryeowook Choi, Soongyu Park, Jongse Lee, Jinwon Kim, Joo-Young Hardware Architecture The explosive arrival of OpenAI's ChatGPT has fueled the globalization of large language model (LLM), which consists of billions of pretrained parameters that embodies the aspects of syntax and semantics. HyperAccel introduces latency processing unit (LPU), a latency-optimized and highly scalable processor architecture for the acceleration of LLM inference. LPU perfectly balances the memory bandwidth and compute logic with streamlined dataflow to maximize performance and efficiency. LPU is equipped with expandable synchronization link (ESL) that hides data synchronization latency between multiple LPUs. HyperDex complements LPU as an intuitive software framework to run LLM applications. LPU achieves 1.25 ms/token and 20.9 ms/token for 1.3B and 66B model, respectively, which is 2.09x and 1.37x faster than the GPU. LPU, synthesized using Samsung 4nm process, has total area of 0.824 mm2 and power consumption of 284.31 mW. LPU-based servers achieve 1.33x and 1.32x energy efficiency over NVIDIA H100 and L4 servers, respectively. |
| title | LPU: A Latency-Optimized and Highly Scalable Processor for Large Language Model Inference |
| topic | Hardware Architecture |
| url | https://arxiv.org/abs/2408.07326 |