Data Management System Analysis for Distributed Computing Workloads
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911187774472192 |
|---|---|
| author | Hsu, Kuan-Chieh Vatsavai, Sairam Sri Kilic, Ozgur O. Korchuganova, Tatiana Nilsson, Paul Dutta, Sankha Ren, Yihui Park, David K. Boudreau, Joseph Chowdhury, Tasnuva Feng, Shengyu Khan, Raees Kim, Jaehyung Klasky, Scott Maeno, Tadashi Outschoorn, Verena Ingrid Martinez Podhorszki, Norbert Suter, Frédéric Yang, Wei Yang, Yiming Yoo, Shinjae Klimentov, Alexei Hoisie, Adolfy |
| author_facet | Hsu, Kuan-Chieh Vatsavai, Sairam Sri Kilic, Ozgur O. Korchuganova, Tatiana Nilsson, Paul Dutta, Sankha Ren, Yihui Park, David K. Boudreau, Joseph Chowdhury, Tasnuva Feng, Shengyu Khan, Raees Kim, Jaehyung Klasky, Scott Maeno, Tadashi Outschoorn, Verena Ingrid Martinez Podhorszki, Norbert Suter, Frédéric Yang, Wei Yang, Yiming Yoo, Shinjae Klimentov, Alexei Hoisie, Adolfy |
| contents | Large-scale international collaborations such as ATLAS rely on globally distributed workflows and data management to process, move, and store vast volumes of data. ATLAS's Production and Distributed Analysis (PanDA) workflow system and the Rucio data management system are each highly optimized for their respective design goals. However, operating them together at global scale exposes systemic inefficiencies, including underutilized resources, redundant or unnecessary transfers, and altered error distributions. Moreover, PanDA and Rucio currently lack shared performance awareness and coordinated, adaptive strategies.
This work charts a path toward co-optimizing the two systems by diagnosing data-management pitfalls and prioritizing end-to-end improvements. With the observation of spatially and temporally imbalanced transfer activities, we develop a metadata-matching algorithm that links PanDA jobs and Rucio datasets at the file level, yielding a complete, fine-grained view of data access and movement. Using this linkage, we identify anomalous transfer patterns that violate PanDA's data-centric job-allocation principle. We then outline mitigation strategies for these patterns and highlight opportunities for tighter PanDA-Rucio coordination to improve resource utilization, reduce unnecessary data movement, and enhance overall system resilience. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_00828 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Data Management System Analysis for Distributed Computing Workloads Hsu, Kuan-Chieh Vatsavai, Sairam Sri Kilic, Ozgur O. Korchuganova, Tatiana Nilsson, Paul Dutta, Sankha Ren, Yihui Park, David K. Boudreau, Joseph Chowdhury, Tasnuva Feng, Shengyu Khan, Raees Kim, Jaehyung Klasky, Scott Maeno, Tadashi Outschoorn, Verena Ingrid Martinez Podhorszki, Norbert Suter, Frédéric Yang, Wei Yang, Yiming Yoo, Shinjae Klimentov, Alexei Hoisie, Adolfy Distributed, Parallel, and Cluster Computing Large-scale international collaborations such as ATLAS rely on globally distributed workflows and data management to process, move, and store vast volumes of data. ATLAS's Production and Distributed Analysis (PanDA) workflow system and the Rucio data management system are each highly optimized for their respective design goals. However, operating them together at global scale exposes systemic inefficiencies, including underutilized resources, redundant or unnecessary transfers, and altered error distributions. Moreover, PanDA and Rucio currently lack shared performance awareness and coordinated, adaptive strategies. This work charts a path toward co-optimizing the two systems by diagnosing data-management pitfalls and prioritizing end-to-end improvements. With the observation of spatially and temporally imbalanced transfer activities, we develop a metadata-matching algorithm that links PanDA jobs and Rucio datasets at the file level, yielding a complete, fine-grained view of data access and movement. Using this linkage, we identify anomalous transfer patterns that violate PanDA's data-centric job-allocation principle. We then outline mitigation strategies for these patterns and highlight opportunities for tighter PanDA-Rucio coordination to improve resource utilization, reduce unnecessary data movement, and enhance overall system resilience. |
| title | Data Management System Analysis for Distributed Computing Workloads |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2510.00828 |