Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2506.12598 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909649236656128 |
|---|---|
| author | Quach, Ryan Wang, Yidi Jahanshahi, Ali Wong, Daniel Kim, Hyoseung |
| author_facet | Quach, Ryan Wang, Yidi Jahanshahi, Ali Wong, Daniel Kim, Hyoseung |
| contents | As AI inference becomes mainstream, research has begun to focus on improving the energy consumption of inference servers. Inference kernels commonly underutilize a GPU's compute resources and waste power from idling components. To improve utilization and energy efficiency, multiple models can co-locate and share the GPU. However, typical GPU spatial partitioning techniques often experience significant overheads when reconfiguring spatial partitions, which can waste additional energy through repartitioning overheads or non-optimal partition configurations. In this paper, we present ECLIP, a framework to enable low-overhead energy-efficient kernel-wise resource partitioning between co-located inference kernels. ECLIP minimizes repartitioning overheads by pre-allocating pools of CU masked streams and assigns optimal CU assignments to groups of kernels through our resource allocation optimizer. Overall, ECLIP achieves an average of 13% improvement to throughput and 25% improvement to energy efficiency. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_12598 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ECLIP: Energy-efficient and Practical Co-Location of ML Inference on Spatially Partitioned GPUs Quach, Ryan Wang, Yidi Jahanshahi, Ali Wong, Daniel Kim, Hyoseung Systems and Control As AI inference becomes mainstream, research has begun to focus on improving the energy consumption of inference servers. Inference kernels commonly underutilize a GPU's compute resources and waste power from idling components. To improve utilization and energy efficiency, multiple models can co-locate and share the GPU. However, typical GPU spatial partitioning techniques often experience significant overheads when reconfiguring spatial partitions, which can waste additional energy through repartitioning overheads or non-optimal partition configurations. In this paper, we present ECLIP, a framework to enable low-overhead energy-efficient kernel-wise resource partitioning between co-located inference kernels. ECLIP minimizes repartitioning overheads by pre-allocating pools of CU masked streams and assigns optimal CU assignments to groups of kernels through our resource allocation optimizer. Overall, ECLIP achieves an average of 13% improvement to throughput and 25% improvement to energy efficiency. |
| title | ECLIP: Energy-efficient and Practical Co-Location of ML Inference on Spatially Partitioned GPUs |
| topic | Systems and Control |
| url | https://arxiv.org/abs/2506.12598 |