Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2504.17584 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912344352751616 |
|---|---|
| author | Liu, Qingyuan Chen, Liyan Yang, Yanning Wang, Haocheng Du, Dong Mao, Zhigang Jing, Naifeng Xia, Yubin Chen, Haibo |
| author_facet | Liu, Qingyuan Chen, Liyan Yang, Yanning Wang, Haocheng Du, Dong Mao, Zhigang Jing, Naifeng Xia, Yubin Chen, Haibo |
| contents | Large Language Models (LLMs) increasingly require processing long text sequences, but GPU memory limitations force difficult trade-offs between memory capacity and bandwidth. While HBM-based acceleration offers high bandwidth, its capacity remains constrained. Offloading data to host-side DIMMs improves capacity but introduces costly data swapping overhead. We identify that the critical memory bottleneck lies in the decoding phase of multi-head attention (MHA) exclusively, which demands substantial capacity for storing KV caches and high bandwidth for attention computation. Our key insight reveals this operation uniquely aligns with modern DIMM-based processing-in-memory (PIM) architectures, which offers scalability of both capacity and bandwidth.
Based on this observation and insight, we propose L3, a hardware-software co-designed system integrating DIMM-PIM and GPU devices. L3 introduces three innovations: First, hardware redesigns resolve data layout mismatches and computational element mismatches in DIMM-PIM, enhancing LLM inference utilization. Second, communication optimization enables hiding the data transfer overhead with the computation. Third, an adaptive scheduler coordinates GPU-DIMM-PIM operations to maximize parallelism between devices. Evaluations using real-world traces show L3 achieves up to 6.1$\times$ speedup over state-of-the-art HBM-PIM solutions while significantly improving batch sizes. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_17584 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | L3: DIMM-PIM Integrated Architecture and Coordination for Scalable Long-Context LLM Inference Liu, Qingyuan Chen, Liyan Yang, Yanning Wang, Haocheng Du, Dong Mao, Zhigang Jing, Naifeng Xia, Yubin Chen, Haibo Hardware Architecture Machine Learning Large Language Models (LLMs) increasingly require processing long text sequences, but GPU memory limitations force difficult trade-offs between memory capacity and bandwidth. While HBM-based acceleration offers high bandwidth, its capacity remains constrained. Offloading data to host-side DIMMs improves capacity but introduces costly data swapping overhead. We identify that the critical memory bottleneck lies in the decoding phase of multi-head attention (MHA) exclusively, which demands substantial capacity for storing KV caches and high bandwidth for attention computation. Our key insight reveals this operation uniquely aligns with modern DIMM-based processing-in-memory (PIM) architectures, which offers scalability of both capacity and bandwidth. Based on this observation and insight, we propose L3, a hardware-software co-designed system integrating DIMM-PIM and GPU devices. L3 introduces three innovations: First, hardware redesigns resolve data layout mismatches and computational element mismatches in DIMM-PIM, enhancing LLM inference utilization. Second, communication optimization enables hiding the data transfer overhead with the computation. Third, an adaptive scheduler coordinates GPU-DIMM-PIM operations to maximize parallelism between devices. Evaluations using real-world traces show L3 achieves up to 6.1$\times$ speedup over state-of-the-art HBM-PIM solutions while significantly improving batch sizes. |
| title | L3: DIMM-PIM Integrated Architecture and Coordination for Scalable Long-Context LLM Inference |
| topic | Hardware Architecture Machine Learning |
| url | https://arxiv.org/abs/2504.17584 |