Balanced segmentation of CNNs for multi-TPU inference
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912255886491648 |
|---|---|
| author | Villarrubia, Jorge Costero, Luis Igual, Francisco D. Olcoz, Katzalin |
| author_facet | Villarrubia, Jorge Costero, Luis Igual, Francisco D. Olcoz, Katzalin |
| contents | In this paper, we propose different alternatives for convolutional neural networks (CNNs) segmentation, addressing inference processes on computing architectures composed by multiple Edge TPUs. Specifically, we compare the inference performance for a number of state-of-the-art CNN models taking as a reference inference times on one TPU and a compiler-based pipelined inference implementation as provided by the Google's Edge TPU compiler. Departing from a profiled-based segmentation strategy, we provide further refinements to balance the workload across multiple TPUs, leveraging their cooperative computing power, reducing work imbalance and alleviating the memory access bottleneck due to the limited amount of on-chip memory per TPU. The observed performance results compared with a single TPU yield superlinear speedups and accelerations up to 2.60x compared with the segmentation offered by the compiler targeting multiple TPUs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_01035 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Balanced segmentation of CNNs for multi-TPU inference Villarrubia, Jorge Costero, Luis Igual, Francisco D. Olcoz, Katzalin Distributed, Parallel, and Cluster Computing C.1.3; B.8 In this paper, we propose different alternatives for convolutional neural networks (CNNs) segmentation, addressing inference processes on computing architectures composed by multiple Edge TPUs. Specifically, we compare the inference performance for a number of state-of-the-art CNN models taking as a reference inference times on one TPU and a compiler-based pipelined inference implementation as provided by the Google's Edge TPU compiler. Departing from a profiled-based segmentation strategy, we provide further refinements to balance the workload across multiple TPUs, leveraging their cooperative computing power, reducing work imbalance and alleviating the memory access bottleneck due to the limited amount of on-chip memory per TPU. The observed performance results compared with a single TPU yield superlinear speedups and accelerations up to 2.60x compared with the segmentation offered by the compiler targeting multiple TPUs. |
| title | Balanced segmentation of CNNs for multi-TPU inference |
| topic | Distributed, Parallel, and Cluster Computing C.1.3; B.8 |
| url | https://arxiv.org/abs/2503.01035 |