Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2502.04093 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912310162882560 |
|---|---|
| author | Yang, Zhuoxun Di, Sheng Zhang, Longtao Li, Ruoyu Li, Ximiao Huang, Jiajun Liu, Jinyang Cappello, Franck Zhao, Kai |
| author_facet | Yang, Zhuoxun Di, Sheng Zhang, Longtao Li, Ruoyu Li, Ximiao Huang, Jiajun Liu, Jinyang Cappello, Franck Zhao, Kai |
| contents | Compression is a crucial solution for data reduction in modern scientific applications due to the exponential growth of data from simulations, experiments, and observations. Compression with progressive retrieval capability allows users to access coarse approximations of data quickly and then incrementally refine these approximations to higher fidelity. Existing progressive compression solutions suffer from low reduction ratios or high operation costs, effectively undermining the approach's benefits. In this paper, we propose the first-ever interpolation-based progressive lossy compression solution that has both high reduction ratios and low operation costs. The interpolation-based algorithm has been verified as one of the best for scientific data reduction, but previously no effort exists to make it support progressive retrieval. Our contributions are three-fold: (1) We thoroughly analyze the error characteristics of the interpolation algorithm and propose our solution IPComp with multi-level bitplane and predictive coding. (2) We derive optimized strategies toward minimum data retrieval under different fidelity levels indicated by users through error bounds and bitrates. (3) We evaluate the proposed solution using six real-world datasets from four diverse domains. Experimental results demonstrate our solution archives up to $487\%$ higher compression ratios and $698\%$ faster speed than other state-of-the-art progressive compressors, and reduces the data volume for retrieval by up to $83\%$ compared to baselines under the same error bound, and reduces the error by up to $99\%$ under the same bitrate. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_04093 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | IPComp: Interpolation Based Progressive Lossy Compression for Scientific Applications Yang, Zhuoxun Di, Sheng Zhang, Longtao Li, Ruoyu Li, Ximiao Huang, Jiajun Liu, Jinyang Cappello, Franck Zhao, Kai Distributed, Parallel, and Cluster Computing Compression is a crucial solution for data reduction in modern scientific applications due to the exponential growth of data from simulations, experiments, and observations. Compression with progressive retrieval capability allows users to access coarse approximations of data quickly and then incrementally refine these approximations to higher fidelity. Existing progressive compression solutions suffer from low reduction ratios or high operation costs, effectively undermining the approach's benefits. In this paper, we propose the first-ever interpolation-based progressive lossy compression solution that has both high reduction ratios and low operation costs. The interpolation-based algorithm has been verified as one of the best for scientific data reduction, but previously no effort exists to make it support progressive retrieval. Our contributions are three-fold: (1) We thoroughly analyze the error characteristics of the interpolation algorithm and propose our solution IPComp with multi-level bitplane and predictive coding. (2) We derive optimized strategies toward minimum data retrieval under different fidelity levels indicated by users through error bounds and bitrates. (3) We evaluate the proposed solution using six real-world datasets from four diverse domains. Experimental results demonstrate our solution archives up to $487\%$ higher compression ratios and $698\%$ faster speed than other state-of-the-art progressive compressors, and reduces the data volume for retrieval by up to $83\%$ compared to baselines under the same error bound, and reduces the error by up to $99\%$ under the same bitrate. |
| title | IPComp: Interpolation Based Progressive Lossy Compression for Scientific Applications |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2502.04093 |