In-Transit Data Transport Strategies for Coupled AI-Simulation Workflow Patterns
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916964764483584 |
|---|---|
| author | Tummalapalli, Harikrishna Balin, Riccardo Simpson, Christine M. Park, Andrew Alsaadi, Aymen Shao, Andrew E. Brewer, Wesley Jha, Shantenu |
| author_facet | Tummalapalli, Harikrishna Balin, Riccardo Simpson, Christine M. Park, Andrew Alsaadi, Aymen Shao, Andrew E. Brewer, Wesley Jha, Shantenu |
| contents | Coupled AI-Simulation workflows are becoming the major workloads for HPC facilities, and their increasing complexity necessitates new tools for performance analysis and prototyping of new in-situ workflows. We present SimAI-Bench, a tool designed to both prototype and evaluate these coupled workflows. In this paper, we use SimAI-Bench to benchmark the data transport performance of two common patterns on the Aurora supercomputer: a one-to-one workflow with co-located simulation and AI training instances, and a many-to-one workflow where a single AI model is trained from an ensemble of simulations. For the one-to-one pattern, our analysis shows that node-local and DragonHPC data staging strategies provide excellent performance compared Redis and Lustre file system. For the many-to-one pattern, we find that data transport becomes a dominant bottleneck as the ensemble size grows. Our evaluation reveals that file system is the optimal solution among the tested strategies for the many-to-one pattern. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_19150 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | In-Transit Data Transport Strategies for Coupled AI-Simulation Workflow Patterns Tummalapalli, Harikrishna Balin, Riccardo Simpson, Christine M. Park, Andrew Alsaadi, Aymen Shao, Andrew E. Brewer, Wesley Jha, Shantenu Distributed, Parallel, and Cluster Computing Coupled AI-Simulation workflows are becoming the major workloads for HPC facilities, and their increasing complexity necessitates new tools for performance analysis and prototyping of new in-situ workflows. We present SimAI-Bench, a tool designed to both prototype and evaluate these coupled workflows. In this paper, we use SimAI-Bench to benchmark the data transport performance of two common patterns on the Aurora supercomputer: a one-to-one workflow with co-located simulation and AI training instances, and a many-to-one workflow where a single AI model is trained from an ensemble of simulations. For the one-to-one pattern, our analysis shows that node-local and DragonHPC data staging strategies provide excellent performance compared Redis and Lustre file system. For the many-to-one pattern, we find that data transport becomes a dominant bottleneck as the ensemble size grows. Our evaluation reveals that file system is the optimal solution among the tested strategies for the many-to-one pattern. |
| title | In-Transit Data Transport Strategies for Coupled AI-Simulation Workflow Patterns |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2509.19150 |