Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2406.18112 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914849241432064 |
|---|---|
| author | Mazen, François Gombert, Louis Givord, Lucas Gueunet, Charles |
| author_facet | Mazen, François Gombert, Louis Givord, Lucas Gueunet, Charles |
| contents | In this short paper, we present an innovative approach to limit the required bandwidth when transferring data during in transit analysis. This approach is called hybrid because it combines existing in situ and in transit solutions. It leverages the stable ABI of Catalyst version 2 and the Catalyst-ADIOS2 implementation to seamlessly switch from in situ, in transit and hybrid analysis without modifying the numerical simulation code. The typical use case is to perform data reduction in situ then generate a visualization in transit on the reduced data. This approach makes the numerical simulation workflows very flexible depending on the size of the data, the available computing resources or the analysis type. Our experiment with this hybrid approach, reducing data before sending it, demonstrated large cost reductions for some visualization pipelines compared to in situ and in transit solutions. The implementation is available under an open source permissive license to be usable broadly in any scientific community. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_18112 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | In Situ In Transit Hybrid Analysis with Catalyst-ADIOS2 Mazen, François Gombert, Louis Givord, Lucas Gueunet, Charles Distributed, Parallel, and Cluster Computing In this short paper, we present an innovative approach to limit the required bandwidth when transferring data during in transit analysis. This approach is called hybrid because it combines existing in situ and in transit solutions. It leverages the stable ABI of Catalyst version 2 and the Catalyst-ADIOS2 implementation to seamlessly switch from in situ, in transit and hybrid analysis without modifying the numerical simulation code. The typical use case is to perform data reduction in situ then generate a visualization in transit on the reduced data. This approach makes the numerical simulation workflows very flexible depending on the size of the data, the available computing resources or the analysis type. Our experiment with this hybrid approach, reducing data before sending it, demonstrated large cost reductions for some visualization pipelines compared to in situ and in transit solutions. The implementation is available under an open source permissive license to be usable broadly in any scientific community. |
| title | In Situ In Transit Hybrid Analysis with Catalyst-ADIOS2 |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2406.18112 |