Measuring Heterogeneity in Machine Learning with Distributed Energy Distance
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913667016032256 |
|---|---|
| author | Fan, Mengchen Geng, Baocheng Shterenberg, Roman Casey, Joseph A. Chen, Zhong Li, Keren |
| author_facet | Fan, Mengchen Geng, Baocheng Shterenberg, Roman Casey, Joseph A. Chen, Zhong Li, Keren |
| contents | In distributed and federated learning, heterogeneity across data sources remains a major obstacle to effective model aggregation and convergence. We focus on feature heterogeneity and introduce energy distance as a sensitive measure for quantifying distributional discrepancies. While we show that energy distance is robust for detecting data distribution shifts, its direct use in large-scale systems can be prohibitively expensive. To address this, we develop Taylor approximations that preserve key theoretical quantitative properties while reducing computational overhead. Through simulation studies, we show how accurately capturing feature discrepancies boosts convergence in distributed learning. Finally, we propose a novel application of energy distance to assign penalty weights for aligning predictions across heterogeneous nodes, ultimately enhancing coordination in federated and distributed settings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_16174 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Measuring Heterogeneity in Machine Learning with Distributed Energy Distance Fan, Mengchen Geng, Baocheng Shterenberg, Roman Casey, Joseph A. Chen, Zhong Li, Keren Machine Learning Artificial Intelligence Distributed, Parallel, and Cluster Computing In distributed and federated learning, heterogeneity across data sources remains a major obstacle to effective model aggregation and convergence. We focus on feature heterogeneity and introduce energy distance as a sensitive measure for quantifying distributional discrepancies. While we show that energy distance is robust for detecting data distribution shifts, its direct use in large-scale systems can be prohibitively expensive. To address this, we develop Taylor approximations that preserve key theoretical quantitative properties while reducing computational overhead. Through simulation studies, we show how accurately capturing feature discrepancies boosts convergence in distributed learning. Finally, we propose a novel application of energy distance to assign penalty weights for aligning predictions across heterogeneous nodes, ultimately enhancing coordination in federated and distributed settings. |
| title | Measuring Heterogeneity in Machine Learning with Distributed Energy Distance |
| topic | Machine Learning Artificial Intelligence Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2501.16174 |