CGSim: A Simulation Framework for Large Scale Distributed Computing Environment
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912620984926208 |
|---|---|
| author | Vatsavai, Sairam Sri Khan, Raees Hsu, Kuan-Chieh Kilic, Ozgur O. Nilsson, Paul Korchuganova, Tatiana Park, David K. Dutta, Sankha Ren, Yihui Boudreau, Joseph Chowdhury, Tasnuva Feng, Shengyu Kim, Jaehyung Klasky, Scott Maeno, Tadashi Martinez, Verena Ingrid Podhorszki, Norbert Suter, Frédéric Yang, Wei Yang, Yiming Yoo, Shinjae Klimentov, Alexei Hoisie, Adolfy |
| author_facet | Vatsavai, Sairam Sri Khan, Raees Hsu, Kuan-Chieh Kilic, Ozgur O. Nilsson, Paul Korchuganova, Tatiana Park, David K. Dutta, Sankha Ren, Yihui Boudreau, Joseph Chowdhury, Tasnuva Feng, Shengyu Kim, Jaehyung Klasky, Scott Maeno, Tadashi Martinez, Verena Ingrid Podhorszki, Norbert Suter, Frédéric Yang, Wei Yang, Yiming Yoo, Shinjae Klimentov, Alexei Hoisie, Adolfy |
| contents | Large-scale distributed computing infrastructures such as the Worldwide LHC Computing Grid (WLCG) require comprehensive simulation tools for evaluating performance, testing new algorithms, and optimizing resource allocation strategies. However, existing simulators suffer from limited scalability, hardwired algorithms, lack of real-time monitoring, and inability to generate datasets suitable for modern machine learning approaches. We present CGSim, a simulation framework for large-scale distributed computing environments that addresses these limitations. Built upon the validated SimGrid simulation framework, CGSim provides high-level abstractions for modeling heterogeneous grid environments while maintaining accuracy and scalability. Key features include a modular plugin mechanism for testing custom workflow scheduling and data movement policies, interactive real-time visualization dashboards, and automatic generation of event-level datasets suitable for AI-assisted performance modeling. We demonstrate CGSim's capabilities through a comprehensive evaluation using production ATLAS PanDA workloads, showing significant calibration accuracy improvements across WLCG computing sites. Scalability experiments show near-linear scaling for multi-site simulations, with distributed workloads achieving 6x better performance compared to single-site execution. The framework enables researchers to simulate WLCG-scale infrastructures with hundreds of sites and thousands of concurrent jobs within practical time budget constraints on commodity hardware. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_00822 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | CGSim: A Simulation Framework for Large Scale Distributed Computing Environment Vatsavai, Sairam Sri Khan, Raees Hsu, Kuan-Chieh Kilic, Ozgur O. Nilsson, Paul Korchuganova, Tatiana Park, David K. Dutta, Sankha Ren, Yihui Boudreau, Joseph Chowdhury, Tasnuva Feng, Shengyu Kim, Jaehyung Klasky, Scott Maeno, Tadashi Martinez, Verena Ingrid Podhorszki, Norbert Suter, Frédéric Yang, Wei Yang, Yiming Yoo, Shinjae Klimentov, Alexei Hoisie, Adolfy Distributed, Parallel, and Cluster Computing Performance Large-scale distributed computing infrastructures such as the Worldwide LHC Computing Grid (WLCG) require comprehensive simulation tools for evaluating performance, testing new algorithms, and optimizing resource allocation strategies. However, existing simulators suffer from limited scalability, hardwired algorithms, lack of real-time monitoring, and inability to generate datasets suitable for modern machine learning approaches. We present CGSim, a simulation framework for large-scale distributed computing environments that addresses these limitations. Built upon the validated SimGrid simulation framework, CGSim provides high-level abstractions for modeling heterogeneous grid environments while maintaining accuracy and scalability. Key features include a modular plugin mechanism for testing custom workflow scheduling and data movement policies, interactive real-time visualization dashboards, and automatic generation of event-level datasets suitable for AI-assisted performance modeling. We demonstrate CGSim's capabilities through a comprehensive evaluation using production ATLAS PanDA workloads, showing significant calibration accuracy improvements across WLCG computing sites. Scalability experiments show near-linear scaling for multi-site simulations, with distributed workloads achieving 6x better performance compared to single-site execution. The framework enables researchers to simulate WLCG-scale infrastructures with hundreds of sites and thousands of concurrent jobs within practical time budget constraints on commodity hardware. |
| title | CGSim: A Simulation Framework for Large Scale Distributed Computing Environment |
| topic | Distributed, Parallel, and Cluster Computing Performance |
| url | https://arxiv.org/abs/2510.00822 |