Parallel Track Transformers: Enabling Fast GPU Inference with Reduced Synchronization
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911430515621888 |
|---|---|
| author | Wang, Chong Du, Nan Gunter, Tom Lei, Tao Seth, Kulin Tong, Senyu Wang, Jianyu Yin, Guoli Zhou, Xiyou Zou, Kelvin Pang, Ruoming |
| author_facet | Wang, Chong Du, Nan Gunter, Tom Lei, Tao Seth, Kulin Tong, Senyu Wang, Jianyu Yin, Guoli Zhou, Xiyou Zou, Kelvin Pang, Ruoming |
| contents | Efficient large-scale inference of transformer-based large language models (LLMs) remains a fundamental systems challenge, frequently requiring multi-GPU parallelism to meet stringent latency and throughput targets. Conventional tensor parallelism decomposes matrix operations across devices but introduces substantial inter-GPU synchronization, leading to communication bottlenecks and degraded scalability. We propose the Parallel Track (PT) Transformer, a novel architectural paradigm that restructures computation to minimize cross-device dependencies. PT achieves up to a 16x reduction in synchronization operations relative to standard tensor parallelism, while maintaining competitive model quality in our experiments. We integrate PT into two widely adopted LLM serving stacks-Tensor-RT-LLM and vLLM-and report consistent improvements in serving efficiency, including up to 15-30% reduced time to first token, 2-12% reduced time per output token, and up to 31.90% increased throughput in both settings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_07306 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Parallel Track Transformers: Enabling Fast GPU Inference with Reduced Synchronization Wang, Chong Du, Nan Gunter, Tom Lei, Tao Seth, Kulin Tong, Senyu Wang, Jianyu Yin, Guoli Zhou, Xiyou Zou, Kelvin Pang, Ruoming Distributed, Parallel, and Cluster Computing Machine Learning Efficient large-scale inference of transformer-based large language models (LLMs) remains a fundamental systems challenge, frequently requiring multi-GPU parallelism to meet stringent latency and throughput targets. Conventional tensor parallelism decomposes matrix operations across devices but introduces substantial inter-GPU synchronization, leading to communication bottlenecks and degraded scalability. We propose the Parallel Track (PT) Transformer, a novel architectural paradigm that restructures computation to minimize cross-device dependencies. PT achieves up to a 16x reduction in synchronization operations relative to standard tensor parallelism, while maintaining competitive model quality in our experiments. We integrate PT into two widely adopted LLM serving stacks-Tensor-RT-LLM and vLLM-and report consistent improvements in serving efficiency, including up to 15-30% reduced time to first token, 2-12% reduced time per output token, and up to 31.90% increased throughput in both settings. |
| title | Parallel Track Transformers: Enabling Fast GPU Inference with Reduced Synchronization |
| topic | Distributed, Parallel, and Cluster Computing Machine Learning |
| url | https://arxiv.org/abs/2602.07306 |