MultiPath Memory Access: Breaking Host-GPU Bandwidth Bottlenecks in LLM Services
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911678363336704 |
|---|---|
| author | Tang, Lingfeng Zhang, Daoping Chen, Junjie Huang, Peihao Jin, Feng Xu, Chengguang Chen, Yuxin Sun, Feiqiang Chen, Guo |
| author_facet | Tang, Lingfeng Zhang, Daoping Chen, Junjie Huang, Peihao Jin, Feng Xu, Chengguang Chen, Yuxin Sun, Feiqiang Chen, Guo |
| contents | Host-GPU data movement has become a latency-critical bottleneck in LLM serving, surfacing in common paths such as model-weight movement and KV cache offload/fetch. Today, each host-GPU copy is effectively confined to the PCIe path of the target GPU, even though modern multi-GPU servers contain additional PCIe links on peer GPUs and high bandwidth GPU interconnects. This leaves substantial intra-server I/O capacity unused. To address this issue, we present Multipath Memory Access (MMA), a software-defined multipath memory access system for host--GPU data transfer. To the best of our knowledge, MMA is the first software-defined system to enable efficient multipath host--GPU data transfer within a single multi-GPU server. MMA expands a single host--GPU copy across available direct and relay paths without hardware, driver, or application changes. It preserves CUDA stream semantics with a dependency-preserving Dummy Task, coordinates distributed micro-transfer completion through a lightweight synchronization mechanism, and uses queue backpressure to route traffic without explicit link-state feedback. On an 8-GPU NVIDIA H20 server, MMA achieves 245 GB/s peak host-to-GPU bandwidth, a 4.62x improvement over native CUDA copies, and reduces TTFT for KV cache fetching by 1.14-2.38x and model wake-up/switching latency by 1.12-2.48x. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_16056 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MultiPath Memory Access: Breaking Host-GPU Bandwidth Bottlenecks in LLM Services Tang, Lingfeng Zhang, Daoping Chen, Junjie Huang, Peihao Jin, Feng Xu, Chengguang Chen, Yuxin Sun, Feiqiang Chen, Guo Distributed, Parallel, and Cluster Computing Networking and Internet Architecture Performance Host-GPU data movement has become a latency-critical bottleneck in LLM serving, surfacing in common paths such as model-weight movement and KV cache offload/fetch. Today, each host-GPU copy is effectively confined to the PCIe path of the target GPU, even though modern multi-GPU servers contain additional PCIe links on peer GPUs and high bandwidth GPU interconnects. This leaves substantial intra-server I/O capacity unused. To address this issue, we present Multipath Memory Access (MMA), a software-defined multipath memory access system for host--GPU data transfer. To the best of our knowledge, MMA is the first software-defined system to enable efficient multipath host--GPU data transfer within a single multi-GPU server. MMA expands a single host--GPU copy across available direct and relay paths without hardware, driver, or application changes. It preserves CUDA stream semantics with a dependency-preserving Dummy Task, coordinates distributed micro-transfer completion through a lightweight synchronization mechanism, and uses queue backpressure to route traffic without explicit link-state feedback. On an 8-GPU NVIDIA H20 server, MMA achieves 245 GB/s peak host-to-GPU bandwidth, a 4.62x improvement over native CUDA copies, and reduces TTFT for KV cache fetching by 1.14-2.38x and model wake-up/switching latency by 1.12-2.48x. |
| title | MultiPath Memory Access: Breaking Host-GPU Bandwidth Bottlenecks in LLM Services |
| topic | Distributed, Parallel, and Cluster Computing Networking and Internet Architecture Performance |
| url | https://arxiv.org/abs/2512.16056 |