SpecInF: Exploiting Idle GPU Resources in Distributed DL Training via Speculative Inference Filling
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916662647717888 |
|---|---|
| author | Lv, Cunchi Shi, Xiao Liang, Dong Tan, Wenting Zhao, Xiaofang |
| author_facet | Lv, Cunchi Shi, Xiao Liang, Dong Tan, Wenting Zhao, Xiaofang |
| contents | Deep Learning (DL), especially with Large Language Models (LLMs), brings benefits to various areas. However, DL training systems usually yield prominent idling GPU resources due to many factors, such as resource allocation and collective communication. To improve GPU utilization, we present SpecInF, which adopts a Speculative Inference Filling method to exploit idle GPU resources. It collocates each primary training instance with additional inference instances on the same GPU, detects the training bubbles and adaptively fills with online or offline inference workloads. Our results show that SpecInF can effectively enhance GPU utilization under mainstream parallel training modes, delivering additional up to 14$\times$ offline inference throughputs than TGS and 67\% reduction in online inference p95 latency than MPS, while guaranteeing collocated training throughput. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_02550 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SpecInF: Exploiting Idle GPU Resources in Distributed DL Training via Speculative Inference Filling Lv, Cunchi Shi, Xiao Liang, Dong Tan, Wenting Zhao, Xiaofang Distributed, Parallel, and Cluster Computing Deep Learning (DL), especially with Large Language Models (LLMs), brings benefits to various areas. However, DL training systems usually yield prominent idling GPU resources due to many factors, such as resource allocation and collective communication. To improve GPU utilization, we present SpecInF, which adopts a Speculative Inference Filling method to exploit idle GPU resources. It collocates each primary training instance with additional inference instances on the same GPU, detects the training bubbles and adaptively fills with online or offline inference workloads. Our results show that SpecInF can effectively enhance GPU utilization under mainstream parallel training modes, delivering additional up to 14$\times$ offline inference throughputs than TGS and 67\% reduction in online inference p95 latency than MPS, while guaranteeing collocated training throughput. |
| title | SpecInF: Exploiting Idle GPU Resources in Distributed DL Training via Speculative Inference Filling |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2503.02550 |