Profiling Concurrent Vision Inference Workloads on NVIDIA Jetson -- Extended
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916892539617280 |
|---|---|
| author | Chakraborty, Abhinaba Tavernier, Wouter Kourtis, Akis Pickavet, Mario Oikonomakis, Andreas Colle, Didier |
| author_facet | Chakraborty, Abhinaba Tavernier, Wouter Kourtis, Akis Pickavet, Mario Oikonomakis, Andreas Colle, Didier |
| contents | The proliferation of IoT devices and advancements in network technologies have intensified the demand for real-time data processing at the network edge. To address these demands, low-power AI accelerators, particularly GPUs, are increasingly deployed for inference tasks, enabling efficient computation while mitigating cloud-based systems' latency and bandwidth limitations. Despite their growing deployment, GPUs remain underutilised even in computationally intensive workloads. This underutilisation stems from the limited understanding of GPU resource sharing, particularly in edge computing scenarios. In this work, we conduct a detailed analysis of both high- and low-level metrics, including GPU utilisation, memory usage, streaming multiprocessor (SM) utilisation, and tensor core usage, to identify bottlenecks and guide hardware-aware optimisations. By integrating traces from multiple profiling tools, we provide a comprehensive view of resource behaviour on NVIDIA Jetson edge devices under concurrent vision inference workloads. Our findings indicate that while GPU utilisation can reach $100\%$ under specific optimisations, critical low-level resources, such as SMs and tensor cores, often operate only at $15\%$ to $30\%$ utilisation. Moreover, we observe that certain CPU-side events, such as thread scheduling, context switching, etc., frequently emerge as bottlenecks, further constraining overall GPU performance. We provide several key observations for users of vision inference workloads on NVIDIA edge devices. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_08430 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Profiling Concurrent Vision Inference Workloads on NVIDIA Jetson -- Extended Chakraborty, Abhinaba Tavernier, Wouter Kourtis, Akis Pickavet, Mario Oikonomakis, Andreas Colle, Didier Distributed, Parallel, and Cluster Computing Hardware Architecture Performance The proliferation of IoT devices and advancements in network technologies have intensified the demand for real-time data processing at the network edge. To address these demands, low-power AI accelerators, particularly GPUs, are increasingly deployed for inference tasks, enabling efficient computation while mitigating cloud-based systems' latency and bandwidth limitations. Despite their growing deployment, GPUs remain underutilised even in computationally intensive workloads. This underutilisation stems from the limited understanding of GPU resource sharing, particularly in edge computing scenarios. In this work, we conduct a detailed analysis of both high- and low-level metrics, including GPU utilisation, memory usage, streaming multiprocessor (SM) utilisation, and tensor core usage, to identify bottlenecks and guide hardware-aware optimisations. By integrating traces from multiple profiling tools, we provide a comprehensive view of resource behaviour on NVIDIA Jetson edge devices under concurrent vision inference workloads. Our findings indicate that while GPU utilisation can reach $100\%$ under specific optimisations, critical low-level resources, such as SMs and tensor cores, often operate only at $15\%$ to $30\%$ utilisation. Moreover, we observe that certain CPU-side events, such as thread scheduling, context switching, etc., frequently emerge as bottlenecks, further constraining overall GPU performance. We provide several key observations for users of vision inference workloads on NVIDIA edge devices. |
| title | Profiling Concurrent Vision Inference Workloads on NVIDIA Jetson -- Extended |
| topic | Distributed, Parallel, and Cluster Computing Hardware Architecture Performance |
| url | https://arxiv.org/abs/2508.08430 |