A Novel Coded Computing Approach for Distributed Multi-Task Learning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909703218397184 |
|---|---|
| author | Cheng, Minquan Wang, Yongkang Zhang, Lingyu Wu, Youlong |
| author_facet | Cheng, Minquan Wang, Yongkang Zhang, Lingyu Wu, Youlong |
| contents | Distributed multi-task learning (DMTL) effectively improves model generalization performance through the collaborative training of multiple related models. However, in large-scale learning scenarios, communication bottlenecks severely limit practical system performance. In this paper, we investigate the communication bottleneck within a typical DMTL system that employs non-linear global updates. This system involves distributed workers, assisted by a central server, who collaboratively learn distinct models derived from a non-linear aggregation of their local model parameters. We first characterize the communication process as a matrix decomposition problem. It transforms workers' data storage constraints into structural characteristics of the uplink encoding matrix, and worker data retrieval demands into Maximum Distance Separable (MDS) properties of the downlink encoding matrix. Building on this, we propose a novel coded DTML scheme that can greatly reduce the communication cost of the DTML with heterogeneous data placement. Theoretical analysis demonstrates that the proposed scheme achieves the theoretical lower bound for communication overhead under mild conditions. Remarkably, this optimality holds for both traditional homogeneous computing environments and various heterogeneous scenarios. Furthermore, our scheme is extensible to a distributed linearly separable computation problem where the target function involves multiple linear combinations of local update values. This indicates that our scheme offers a new way of tackling heterogeneous data placement challenges in various distributed applications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_18025 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | A Novel Coded Computing Approach for Distributed Multi-Task Learning Cheng, Minquan Wang, Yongkang Zhang, Lingyu Wu, Youlong Information Theory Distributed multi-task learning (DMTL) effectively improves model generalization performance through the collaborative training of multiple related models. However, in large-scale learning scenarios, communication bottlenecks severely limit practical system performance. In this paper, we investigate the communication bottleneck within a typical DMTL system that employs non-linear global updates. This system involves distributed workers, assisted by a central server, who collaboratively learn distinct models derived from a non-linear aggregation of their local model parameters. We first characterize the communication process as a matrix decomposition problem. It transforms workers' data storage constraints into structural characteristics of the uplink encoding matrix, and worker data retrieval demands into Maximum Distance Separable (MDS) properties of the downlink encoding matrix. Building on this, we propose a novel coded DTML scheme that can greatly reduce the communication cost of the DTML with heterogeneous data placement. Theoretical analysis demonstrates that the proposed scheme achieves the theoretical lower bound for communication overhead under mild conditions. Remarkably, this optimality holds for both traditional homogeneous computing environments and various heterogeneous scenarios. Furthermore, our scheme is extensible to a distributed linearly separable computation problem where the target function involves multiple linear combinations of local update values. This indicates that our scheme offers a new way of tackling heterogeneous data placement challenges in various distributed applications. |
| title | A Novel Coded Computing Approach for Distributed Multi-Task Learning |
| topic | Information Theory |
| url | https://arxiv.org/abs/2507.18025 |