Improving inference time in multi-TPU systems with profiled model segmentation
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866912255870763008 |
|---|---|
| author | Villarrubia, Jorge Costero, Luis Igual, Francisco D. Olcoz, Katzalin |
| author_facet | Villarrubia, Jorge Costero, Luis Igual, Francisco D. Olcoz, Katzalin |
| contents | In this paper, we systematically evaluate the inference performance of the Edge TPU by Google for neural networks with different characteristics. Specifically, we determine that, given the limited amount of on-chip memory on the Edge TPU, accesses to external (host) memory rapidly become an important performance bottleneck. We demonstrate how multiple devices can be jointly used to alleviate the bottleneck introduced by accessing the host memory. We propose a solution combining model segmentation and pipelining on up to four TPUs, with remarkable performance improvements that range from $6\times$ for neural networks with convolutional layers to $46\times$ for fully connected layers, compared with single-TPU setups. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_01025 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Improving inference time in multi-TPU systems with profiled model segmentation Villarrubia, Jorge Costero, Luis Igual, Francisco D. Olcoz, Katzalin Distributed, Parallel, and Cluster Computing C.1.3; B.8 In this paper, we systematically evaluate the inference performance of the Edge TPU by Google for neural networks with different characteristics. Specifically, we determine that, given the limited amount of on-chip memory on the Edge TPU, accesses to external (host) memory rapidly become an important performance bottleneck. We demonstrate how multiple devices can be jointly used to alleviate the bottleneck introduced by accessing the host memory. We propose a solution combining model segmentation and pipelining on up to four TPUs, with remarkable performance improvements that range from $6\times$ for neural networks with convolutional layers to $46\times$ for fully connected layers, compared with single-TPU setups. |
| title | Improving inference time in multi-TPU systems with profiled model segmentation |
| topic | Distributed, Parallel, and Cluster Computing C.1.3; B.8 |
| url | https://arxiv.org/abs/2503.01025 |