PIMphony: Overcoming Bandwidth and Capacity Inefficiency in PIM-based Long-Context LLM Inference System
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912788587216896 |
|---|---|
| author | Kwon, Hyucksung Koo, Kyungmo Kim, Janghyeon Lee, Woongkyu Lee, Minjae Jung, Gyeonggeun Lee, Hyungdeok Jung, Yousub Park, Jaehan Song, Yosub Yang, Byeongsu Choi, Haerang Kim, Guhyun Won, Jongsoon Shin, Woojae Kim, Changhyun Shin, Gyeongcheol Kwon, Yongkee Kim, Ilkon Lim, Euicheol Kim, John Choi, Jungwook |
| author_facet | Kwon, Hyucksung Koo, Kyungmo Kim, Janghyeon Lee, Woongkyu Lee, Minjae Jung, Gyeonggeun Lee, Hyungdeok Jung, Yousub Park, Jaehan Song, Yosub Yang, Byeongsu Choi, Haerang Kim, Guhyun Won, Jongsoon Shin, Woojae Kim, Changhyun Shin, Gyeongcheol Kwon, Yongkee Kim, Ilkon Lim, Euicheol Kim, John Choi, Jungwook |
| contents | The expansion of long-context Large Language Models (LLMs) creates significant memory system challenges. While Processing-in-Memory (PIM) is a promising accelerator, we identify that it suffers from critical inefficiencies when scaled to long contexts: severe channel underutilization, performance-limiting I/O bottlenecks, and massive memory waste from static KV cache management. In this work, we propose PIMphony, a PIM orchestrator that systematically resolves these issues with three co-designed techniques. First, Token-Centric PIM Partitioning (TCP) ensures high channel utilization regardless of batch size. Second, Dynamic PIM Command Scheduling (DCS) mitigates the I/O bottleneck by overlapping data movement and computation. Finally, a Dynamic PIM Access (DPA) controller enables dynamic memory management to eliminate static memory waste. Implemented via an MLIR-based compiler and evaluated on a cycle-accurate simulator, PIMphony significantly improves throughput for long-context LLM inference (up to 72B parameters and 1M context length). Our evaluations show performance boosts of up to 11.3x on PIM-only systems and 8.4x on xPU+PIM systems, enabling more efficient deployment of LLMs in real-world long-context applications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_20166 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | PIMphony: Overcoming Bandwidth and Capacity Inefficiency in PIM-based Long-Context LLM Inference System Kwon, Hyucksung Koo, Kyungmo Kim, Janghyeon Lee, Woongkyu Lee, Minjae Jung, Gyeonggeun Lee, Hyungdeok Jung, Yousub Park, Jaehan Song, Yosub Yang, Byeongsu Choi, Haerang Kim, Guhyun Won, Jongsoon Shin, Woojae Kim, Changhyun Shin, Gyeongcheol Kwon, Yongkee Kim, Ilkon Lim, Euicheol Kim, John Choi, Jungwook Hardware Architecture Artificial Intelligence The expansion of long-context Large Language Models (LLMs) creates significant memory system challenges. While Processing-in-Memory (PIM) is a promising accelerator, we identify that it suffers from critical inefficiencies when scaled to long contexts: severe channel underutilization, performance-limiting I/O bottlenecks, and massive memory waste from static KV cache management. In this work, we propose PIMphony, a PIM orchestrator that systematically resolves these issues with three co-designed techniques. First, Token-Centric PIM Partitioning (TCP) ensures high channel utilization regardless of batch size. Second, Dynamic PIM Command Scheduling (DCS) mitigates the I/O bottleneck by overlapping data movement and computation. Finally, a Dynamic PIM Access (DPA) controller enables dynamic memory management to eliminate static memory waste. Implemented via an MLIR-based compiler and evaluated on a cycle-accurate simulator, PIMphony significantly improves throughput for long-context LLM inference (up to 72B parameters and 1M context length). Our evaluations show performance boosts of up to 11.3x on PIM-only systems and 8.4x on xPU+PIM systems, enabling more efficient deployment of LLMs in real-world long-context applications. |
| title | PIMphony: Overcoming Bandwidth and Capacity Inefficiency in PIM-based Long-Context LLM Inference System |
| topic | Hardware Architecture Artificial Intelligence |
| url | https://arxiv.org/abs/2412.20166 |