Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2503.06322 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917948544778240 |
|---|---|
| author | Chen, Jieyang Gong, Qian Li, Yanliang Liang, Xin Wan, Lipeng Liu, Qing Podhorszki, Norbert Klasky, Scott |
| author_facet | Chen, Jieyang Gong, Qian Li, Yanliang Liang, Xin Wan, Lipeng Liu, Qing Podhorszki, Norbert Klasky, Scott |
| contents | The rapid growth of scientific data is surpassing advancements in computing, creating challenges in storage, transfer, and analysis, particularly at the exascale. While data reduction techniques such as lossless and lossy compression help mitigate these issues, their computational overhead introduces new bottlenecks. GPU-accelerated approaches improve performance but face challenges in portability, memory transfer, and scalability on multi-GPU systems. To address these, we propose HPDR, a high-performance, portable data reduction framework. HPDR supports diverse processor architectures, reducing memory transfer overhead to 2.3% and achieving up to 3.5x faster throughput than existing solutions. It attains 96% of the theoretical speedup in multi-GPU settings. Evaluations on the Frontier supercomputer demonstrate 103 TB/s throughput and up to 4x acceleration in parallel I/O performance at scale. HPDR offers a scalable, efficient solution for managing massive data volumes in exascale computing environments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_06322 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | HPDR: High-Performance Portable Scientific Data Reduction Framework Chen, Jieyang Gong, Qian Li, Yanliang Liang, Xin Wan, Lipeng Liu, Qing Podhorszki, Norbert Klasky, Scott Distributed, Parallel, and Cluster Computing The rapid growth of scientific data is surpassing advancements in computing, creating challenges in storage, transfer, and analysis, particularly at the exascale. While data reduction techniques such as lossless and lossy compression help mitigate these issues, their computational overhead introduces new bottlenecks. GPU-accelerated approaches improve performance but face challenges in portability, memory transfer, and scalability on multi-GPU systems. To address these, we propose HPDR, a high-performance, portable data reduction framework. HPDR supports diverse processor architectures, reducing memory transfer overhead to 2.3% and achieving up to 3.5x faster throughput than existing solutions. It attains 96% of the theoretical speedup in multi-GPU settings. Evaluations on the Frontier supercomputer demonstrate 103 TB/s throughput and up to 4x acceleration in parallel I/O performance at scale. HPDR offers a scalable, efficient solution for managing massive data volumes in exascale computing environments. |
| title | HPDR: High-Performance Portable Scientific Data Reduction Framework |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2503.06322 |