Fulcrum: Optimizing Concurrent DNN Training and Inferencing on Edge Accelerators
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914054019219456 |
|---|---|
| author | K., Prashanthi S. Taluri, Saisamarth Gupta, Pranav Saikia, Amartya Ranjan Sahoo, Kunal Kumar Joshi, Atharva Vinay Karwa, Lakshya Dhule, Kedar Simmhan, Yogesh |
| author_facet | K., Prashanthi S. Taluri, Saisamarth Gupta, Pranav Saikia, Amartya Ranjan Sahoo, Kunal Kumar Joshi, Atharva Vinay Karwa, Lakshya Dhule, Kedar Simmhan, Yogesh |
| contents | The proliferation of GPU accelerated edge devices like Nvidia Jetsons and the rise in privacy concerns are placing an emphasis on concurrent DNN training and inferencing on edge devices. Inference and training have different computing and QoS goals. But edge accelerators like Jetson do not support native GPU sharing and expose 1000s of power modes. This requires careful time-sharing of concurrent workloads to meet power--performance goals, while limiting costly profiling. In this paper, we design an intelligent time-slicing approach for concurrent DNN training and inferencing on Jetsons. We formulate an optimization problem to interleave training and inferencing minibatches, and decide the device power mode and inference minibatch size, while maximizing the training throughput and staying within latency and power budgets, with modest profiling costs. We propose GMD, an efficient multi-dimensional gradient descent search which profiles just $15$ power modes; and ALS, an Active Learning technique which identifies reusable Pareto-optimal power modes, but profiles $50$--$150$ power modes. We evaluate these within our Fulcrum scheduler for $273,000+$ configurations across $15$ DNN workloads. We also evaluate our strategies on dynamic arrival inference and concurrent inferences. ALS and GMD outperform simpler and more complex baselines with larger-scale profiling. Their solutions satisfy the latency and power budget for $>97\%$ of our runs, and on average are within $7\%$ of the optimal throughput. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_20205 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Fulcrum: Optimizing Concurrent DNN Training and Inferencing on Edge Accelerators K., Prashanthi S. Taluri, Saisamarth Gupta, Pranav Saikia, Amartya Ranjan Sahoo, Kunal Kumar Joshi, Atharva Vinay Karwa, Lakshya Dhule, Kedar Simmhan, Yogesh Distributed, Parallel, and Cluster Computing The proliferation of GPU accelerated edge devices like Nvidia Jetsons and the rise in privacy concerns are placing an emphasis on concurrent DNN training and inferencing on edge devices. Inference and training have different computing and QoS goals. But edge accelerators like Jetson do not support native GPU sharing and expose 1000s of power modes. This requires careful time-sharing of concurrent workloads to meet power--performance goals, while limiting costly profiling. In this paper, we design an intelligent time-slicing approach for concurrent DNN training and inferencing on Jetsons. We formulate an optimization problem to interleave training and inferencing minibatches, and decide the device power mode and inference minibatch size, while maximizing the training throughput and staying within latency and power budgets, with modest profiling costs. We propose GMD, an efficient multi-dimensional gradient descent search which profiles just $15$ power modes; and ALS, an Active Learning technique which identifies reusable Pareto-optimal power modes, but profiles $50$--$150$ power modes. We evaluate these within our Fulcrum scheduler for $273,000+$ configurations across $15$ DNN workloads. We also evaluate our strategies on dynamic arrival inference and concurrent inferences. ALS and GMD outperform simpler and more complex baselines with larger-scale profiling. Their solutions satisfy the latency and power budget for $>97\%$ of our runs, and on average are within $7\%$ of the optimal throughput. |
| title | Fulcrum: Optimizing Concurrent DNN Training and Inferencing on Edge Accelerators |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2509.20205 |