AI Surrogate Model for Distributed Computing Workloads
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910644057407488 |
|---|---|
| author | Park, David K. Ren, Yihui Kilic, Ozgur O. Korchuganova, Tatiana Vatsavai, Sairam Sri Boudreau, Joseph Chowdhury, Tasnuva Feng, Shengyu Khan, Raees Kim, Jaehyung Klasky, Scott Maeno, Tadashi Nilsson, Paul Outschoorn, Verena Ingrid Martinez Podhorszki, Norbert Suter, Frederic Yang, Wei Yang, Yiming Yoo, Shinjae Klimentov, Alexei Hoisie, Adolfy |
| author_facet | Park, David K. Ren, Yihui Kilic, Ozgur O. Korchuganova, Tatiana Vatsavai, Sairam Sri Boudreau, Joseph Chowdhury, Tasnuva Feng, Shengyu Khan, Raees Kim, Jaehyung Klasky, Scott Maeno, Tadashi Nilsson, Paul Outschoorn, Verena Ingrid Martinez Podhorszki, Norbert Suter, Frederic Yang, Wei Yang, Yiming Yoo, Shinjae Klimentov, Alexei Hoisie, Adolfy |
| contents | Large-scale international scientific collaborations, such as ATLAS, Belle II, CMS, and DUNE, generate vast volumes of data. These experiments necessitate substantial computational power for varied tasks, including structured data processing, Monte Carlo simulations, and end-user analysis. Centralized workflow and data management systems are employed to handle these demands, but current decision-making processes for data placement and payload allocation are often heuristic and disjointed. This optimization challenge potentially could be addressed using contemporary machine learning methods, such as reinforcement learning, which, in turn, require access to extensive data and an interactive environment. Instead, we propose a generative surrogate modeling approach to address the lack of training data and concerns about privacy preservation. We have collected and processed real-world job submission records, totaling more than two million jobs through 150 days, and applied four generative models for tabular data -- TVAE, CTAGGAN+, SMOTE, and TabDDPM -- to these datasets, thoroughly evaluating their performance. Along with measuring the discrepancy among feature-wise distributions separately, we also evaluate pair-wise feature correlations, distance to closest record, and responses to pre-trained models. Our experiments indicate that SMOTE and TabDDPM can generate similar tabular data, almost indistinguishable from the ground truth. Yet, as a non-learning method, SMOTE ranks the lowest in privacy preservation. As a result, we conclude that the probabilistic-diffusion-model-based TabDDPM is the most suitable generative model for managing job record data. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_07940 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | AI Surrogate Model for Distributed Computing Workloads Park, David K. Ren, Yihui Kilic, Ozgur O. Korchuganova, Tatiana Vatsavai, Sairam Sri Boudreau, Joseph Chowdhury, Tasnuva Feng, Shengyu Khan, Raees Kim, Jaehyung Klasky, Scott Maeno, Tadashi Nilsson, Paul Outschoorn, Verena Ingrid Martinez Podhorszki, Norbert Suter, Frederic Yang, Wei Yang, Yiming Yoo, Shinjae Klimentov, Alexei Hoisie, Adolfy Distributed, Parallel, and Cluster Computing Large-scale international scientific collaborations, such as ATLAS, Belle II, CMS, and DUNE, generate vast volumes of data. These experiments necessitate substantial computational power for varied tasks, including structured data processing, Monte Carlo simulations, and end-user analysis. Centralized workflow and data management systems are employed to handle these demands, but current decision-making processes for data placement and payload allocation are often heuristic and disjointed. This optimization challenge potentially could be addressed using contemporary machine learning methods, such as reinforcement learning, which, in turn, require access to extensive data and an interactive environment. Instead, we propose a generative surrogate modeling approach to address the lack of training data and concerns about privacy preservation. We have collected and processed real-world job submission records, totaling more than two million jobs through 150 days, and applied four generative models for tabular data -- TVAE, CTAGGAN+, SMOTE, and TabDDPM -- to these datasets, thoroughly evaluating their performance. Along with measuring the discrepancy among feature-wise distributions separately, we also evaluate pair-wise feature correlations, distance to closest record, and responses to pre-trained models. Our experiments indicate that SMOTE and TabDDPM can generate similar tabular data, almost indistinguishable from the ground truth. Yet, as a non-learning method, SMOTE ranks the lowest in privacy preservation. As a result, we conclude that the probabilistic-diffusion-model-based TabDDPM is the most suitable generative model for managing job record data. |
| title | AI Surrogate Model for Distributed Computing Workloads |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2410.07940 |