Disaggregated Multi-Tower: Topology-aware Modeling Technique for Efficient Large-Scale Recommendation
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910431008784384 |
|---|---|
| author | Luo, Liang Zhang, Buyun Tsang, Michael Ma, Yinbin Chu, Ching-Hsiang Chen, Yuxin Li, Shen Hao, Yuchen Zhao, Yanli Lakshminarayanan, Guna Wen, Ellie Dingqiao Park, Jongsoo Mudigere, Dheevatsa Naumov, Maxim |
| author_facet | Luo, Liang Zhang, Buyun Tsang, Michael Ma, Yinbin Chu, Ching-Hsiang Chen, Yuxin Li, Shen Hao, Yuchen Zhao, Yanli Lakshminarayanan, Guna Wen, Ellie Dingqiao Park, Jongsoo Mudigere, Dheevatsa Naumov, Maxim |
| contents | We study a mismatch between the deep learning recommendation models' flat architecture, common distributed training paradigm and hierarchical data center topology. To address the associated inefficiencies, we propose Disaggregated Multi-Tower (DMT), a modeling technique that consists of (1) Semantic-preserving Tower Transform (SPTT), a novel training paradigm that decomposes the monolithic global embedding lookup process into disjoint towers to exploit data center locality; (2) Tower Module (TM), a synergistic dense component attached to each tower to reduce model complexity and communication volume through hierarchical feature interaction; and (3) Tower Partitioner (TP), a feature partitioner to systematically create towers with meaningful feature interactions and load balanced assignments to preserve model quality and training throughput via learned embeddings. We show that DMT can achieve up to 1.9x speedup compared to the state-of-the-art baselines without losing accuracy across multiple generations of hardware at large data center scales. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_00877 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Disaggregated Multi-Tower: Topology-aware Modeling Technique for Efficient Large-Scale Recommendation Luo, Liang Zhang, Buyun Tsang, Michael Ma, Yinbin Chu, Ching-Hsiang Chen, Yuxin Li, Shen Hao, Yuchen Zhao, Yanli Lakshminarayanan, Guna Wen, Ellie Dingqiao Park, Jongsoo Mudigere, Dheevatsa Naumov, Maxim Machine Learning Distributed, Parallel, and Cluster Computing Information Retrieval We study a mismatch between the deep learning recommendation models' flat architecture, common distributed training paradigm and hierarchical data center topology. To address the associated inefficiencies, we propose Disaggregated Multi-Tower (DMT), a modeling technique that consists of (1) Semantic-preserving Tower Transform (SPTT), a novel training paradigm that decomposes the monolithic global embedding lookup process into disjoint towers to exploit data center locality; (2) Tower Module (TM), a synergistic dense component attached to each tower to reduce model complexity and communication volume through hierarchical feature interaction; and (3) Tower Partitioner (TP), a feature partitioner to systematically create towers with meaningful feature interactions and load balanced assignments to preserve model quality and training throughput via learned embeddings. We show that DMT can achieve up to 1.9x speedup compared to the state-of-the-art baselines without losing accuracy across multiple generations of hardware at large data center scales. |
| title | Disaggregated Multi-Tower: Topology-aware Modeling Technique for Efficient Large-Scale Recommendation |
| topic | Machine Learning Distributed, Parallel, and Cluster Computing Information Retrieval |
| url | https://arxiv.org/abs/2403.00877 |