Towards an Introspective Dynamic Model of Globally Distributed Computing Infrastructures
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866912447090130944 |
|---|---|
| author | Kilic, Ozgur O. Park, David K. Ren, Yihui Korchuganova, Tatiana Vatsavai, Sairam Sri Boudreau, Joseph Chowdhury, Tasnuva Feng, Shengyu Khan, Raees Kim, Jaehyung Klasky, Scott Maeno, Tadashi Nilsson, Paul Outschoorn, Verena Ingrid Martinez Podhorszki, Norbert Suter, Frédéric Yang, Wei Yang, Yiming Yoo, Shinjae Klimentov, Alexei Hoisie, Adolfy |
| author_facet | Kilic, Ozgur O. Park, David K. Ren, Yihui Korchuganova, Tatiana Vatsavai, Sairam Sri Boudreau, Joseph Chowdhury, Tasnuva Feng, Shengyu Khan, Raees Kim, Jaehyung Klasky, Scott Maeno, Tadashi Nilsson, Paul Outschoorn, Verena Ingrid Martinez Podhorszki, Norbert Suter, Frédéric Yang, Wei Yang, Yiming Yoo, Shinjae Klimentov, Alexei Hoisie, Adolfy |
| contents | Large-scale scientific collaborations like ATLAS, Belle II, CMS, DUNE, and others involve hundreds of research institutes and thousands of researchers spread across the globe. These experiments generate petabytes of data, with volumes soon expected to reach exabytes. Consequently, there is a growing need for computation, including structured data processing from raw data to consumer-ready derived data, extensive Monte Carlo simulation campaigns, and a wide range of end-user analysis. To manage these computational and storage demands, centralized workflow and data management systems are implemented. However, decisions regarding data placement and payload allocation are often made disjointly and via heuristic means. A significant obstacle in adopting more effective heuristic or AI-driven solutions is the absence of a quick and reliable introspective dynamic model to evaluate and refine alternative approaches. In this study, we aim to develop such an interactive system using real-world data. By examining job execution records from the PanDA workflow management system, we have pinpointed key performance indicators such as queuing time, error rate, and the extent of remote data access. The dataset includes five months of activity. Additionally, we are creating a generative AI model to simulate time series of payloads, which incorporate visible features like category, event count, and submitting group, as well as hidden features like the total computational load-derived from existing PanDA records and computing site capabilities. These hidden features, which are not visible to job allocators, whether heuristic or AI-driven, influence factors such as queuing times and data movement. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_19578 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Towards an Introspective Dynamic Model of Globally Distributed Computing Infrastructures Kilic, Ozgur O. Park, David K. Ren, Yihui Korchuganova, Tatiana Vatsavai, Sairam Sri Boudreau, Joseph Chowdhury, Tasnuva Feng, Shengyu Khan, Raees Kim, Jaehyung Klasky, Scott Maeno, Tadashi Nilsson, Paul Outschoorn, Verena Ingrid Martinez Podhorszki, Norbert Suter, Frédéric Yang, Wei Yang, Yiming Yoo, Shinjae Klimentov, Alexei Hoisie, Adolfy Distributed, Parallel, and Cluster Computing Artificial Intelligence Large-scale scientific collaborations like ATLAS, Belle II, CMS, DUNE, and others involve hundreds of research institutes and thousands of researchers spread across the globe. These experiments generate petabytes of data, with volumes soon expected to reach exabytes. Consequently, there is a growing need for computation, including structured data processing from raw data to consumer-ready derived data, extensive Monte Carlo simulation campaigns, and a wide range of end-user analysis. To manage these computational and storage demands, centralized workflow and data management systems are implemented. However, decisions regarding data placement and payload allocation are often made disjointly and via heuristic means. A significant obstacle in adopting more effective heuristic or AI-driven solutions is the absence of a quick and reliable introspective dynamic model to evaluate and refine alternative approaches. In this study, we aim to develop such an interactive system using real-world data. By examining job execution records from the PanDA workflow management system, we have pinpointed key performance indicators such as queuing time, error rate, and the extent of remote data access. The dataset includes five months of activity. Additionally, we are creating a generative AI model to simulate time series of payloads, which incorporate visible features like category, event count, and submitting group, as well as hidden features like the total computational load-derived from existing PanDA records and computing site capabilities. These hidden features, which are not visible to job allocators, whether heuristic or AI-driven, influence factors such as queuing times and data movement. |
| title | Towards an Introspective Dynamic Model of Globally Distributed Computing Infrastructures |
| topic | Distributed, Parallel, and Cluster Computing Artificial Intelligence |
| url | https://arxiv.org/abs/2506.19578 |