Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2504.16112 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908719106752512 |
|---|---|
| author | Rhee, Myunghyun Sim, Joonseop Ahn, Taeyoung Lee, Seungyong Yoon, Daegun Kim, Euiseok Park, Kyoung Joo, Youngpyo Kim, Hoshik |
| author_facet | Rhee, Myunghyun Sim, Joonseop Ahn, Taeyoung Lee, Seungyong Yoon, Daegun Kim, Euiseok Park, Kyoung Joo, Youngpyo Kim, Hoshik |
| contents | The attention layer, a core component of Transformer-based LLMs, brings out inefficiencies in current GPU systems due to its low operational intensity and the substantial memory requirements of KV caches. We propose a High-bandwidth Processing Unit (HPU), a memoryintensive co-processor that enhances GPU resource utilization during large-batched LLM inference. By offloading memory-bound operations, the HPU allows the GPU to focus on compute-intensive tasks, increasing overall efficiency. Also, the HPU, as an add-on card, scales out to accommodate surging memory demands driven by large batch sizes and extended sequence lengths. In this paper, we show the HPU prototype implemented with PCIe-based FPGA cards mounted on a GPU system. Our novel GPU-HPU heterogeneous system demonstrates up to 4.1x performance gains and 4.6x energy efficiency improvements over a GPUonly system, providing scalability without increasing the number of GPUs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_16112 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | HPU: High-Bandwidth Processing Unit for Scalable, Cost-effective LLM Inference via GPU Co-processing Rhee, Myunghyun Sim, Joonseop Ahn, Taeyoung Lee, Seungyong Yoon, Daegun Kim, Euiseok Park, Kyoung Joo, Youngpyo Kim, Hoshik Hardware Architecture Artificial Intelligence Computation and Language Distributed, Parallel, and Cluster Computing The attention layer, a core component of Transformer-based LLMs, brings out inefficiencies in current GPU systems due to its low operational intensity and the substantial memory requirements of KV caches. We propose a High-bandwidth Processing Unit (HPU), a memoryintensive co-processor that enhances GPU resource utilization during large-batched LLM inference. By offloading memory-bound operations, the HPU allows the GPU to focus on compute-intensive tasks, increasing overall efficiency. Also, the HPU, as an add-on card, scales out to accommodate surging memory demands driven by large batch sizes and extended sequence lengths. In this paper, we show the HPU prototype implemented with PCIe-based FPGA cards mounted on a GPU system. Our novel GPU-HPU heterogeneous system demonstrates up to 4.1x performance gains and 4.6x energy efficiency improvements over a GPUonly system, providing scalability without increasing the number of GPUs. |
| title | HPU: High-Bandwidth Processing Unit for Scalable, Cost-effective LLM Inference via GPU Co-processing |
| topic | Hardware Architecture Artificial Intelligence Computation and Language Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2504.16112 |