Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2023
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2306.08586 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910980848484352 |
|---|---|
| author | Farhat, Yehya Shili, Hamza ElMokhtar Liao, Fangshuo Dun, Chen Garcia, Mirian Hipolito Zheng, Guoqing Awadallah, Ahmed Hassan Sim, Robert Dimitriadis, Dimitrios Kyrillidis, Anastasios |
| author_facet | Farhat, Yehya Shili, Hamza ElMokhtar Liao, Fangshuo Dun, Chen Garcia, Mirian Hipolito Zheng, Guoqing Awadallah, Ahmed Hassan Sim, Robert Dimitriadis, Dimitrios Kyrillidis, Anastasios |
| contents | Mixture-of-Experts (MoEs) achieve scalability by dynamically activating subsets of their components. Yet, understanding how expertise emerges through joint training of gating mechanisms and experts remains incomplete, especially in scenarios without clear task partitions. Motivated by inference costs and data heterogeneity, we study how joint training of gating functions and experts can dynamically allocate domain-specific expertise across multiple underlying data distributions. As an outcome of our framework, we develop an instance tailored specifically to decentralized training scenarios, introducing \textit{Dynamically Decentralized Orchestration of MoEs} or \texttt{DDOME}. \texttt{DDOME} leverages heterogeneity emerging from distributional shifts across decentralized data sources to specialize experts dynamically. By integrating a pretrained common expert to inform a gating function, \texttt{DDOME} achieves personalized expert subset selection on-the-fly, facilitating just-in-time personalization. We empirically validate \texttt{DDOME} within a Federated Learning (FL) context: \texttt{DDOME} attains from 4\% up to an 24\% accuracy improvement over state-of-the-art FL baselines in image and text classification tasks, while maintaining competitive zero-shot generalization capabilities. Furthermore, we provide theoretical insights confirming that the joint gating-experts training is critical for achieving meaningful expert specialization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2306_08586 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Learning to Specialize: Joint Gating-Expert Training for Adaptive MoEs in Decentralized Settings Farhat, Yehya Shili, Hamza ElMokhtar Liao, Fangshuo Dun, Chen Garcia, Mirian Hipolito Zheng, Guoqing Awadallah, Ahmed Hassan Sim, Robert Dimitriadis, Dimitrios Kyrillidis, Anastasios Machine Learning Artificial Intelligence Optimization and Control Mixture-of-Experts (MoEs) achieve scalability by dynamically activating subsets of their components. Yet, understanding how expertise emerges through joint training of gating mechanisms and experts remains incomplete, especially in scenarios without clear task partitions. Motivated by inference costs and data heterogeneity, we study how joint training of gating functions and experts can dynamically allocate domain-specific expertise across multiple underlying data distributions. As an outcome of our framework, we develop an instance tailored specifically to decentralized training scenarios, introducing \textit{Dynamically Decentralized Orchestration of MoEs} or \texttt{DDOME}. \texttt{DDOME} leverages heterogeneity emerging from distributional shifts across decentralized data sources to specialize experts dynamically. By integrating a pretrained common expert to inform a gating function, \texttt{DDOME} achieves personalized expert subset selection on-the-fly, facilitating just-in-time personalization. We empirically validate \texttt{DDOME} within a Federated Learning (FL) context: \texttt{DDOME} attains from 4\% up to an 24\% accuracy improvement over state-of-the-art FL baselines in image and text classification tasks, while maintaining competitive zero-shot generalization capabilities. Furthermore, we provide theoretical insights confirming that the joint gating-experts training is critical for achieving meaningful expert specialization. |
| title | Learning to Specialize: Joint Gating-Expert Training for Adaptive MoEs in Decentralized Settings |
| topic | Machine Learning Artificial Intelligence Optimization and Control |
| url | https://arxiv.org/abs/2306.08586 |