STZ: A High Quality and High Speed Streaming Lossy Compression Framework for Scientific Data
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911133156245504 |
|---|---|
| author | Wang, Daoce Grosset, Pascal Pulido, Jesus Tian, Jiannan Athawale, Tushar M. Jia, Jinda Sun, Baixi Zhang, Boyuan Jin, Sian Zhao, Kai Ahrens, James Song, Fengguang |
| author_facet | Wang, Daoce Grosset, Pascal Pulido, Jesus Tian, Jiannan Athawale, Tushar M. Jia, Jinda Sun, Baixi Zhang, Boyuan Jin, Sian Zhao, Kai Ahrens, James Song, Fengguang |
| contents | Error-bounded lossy compression is one of the most efficient solutions to reduce the volume of scientific data. For lossy compression, progressive decompression and random-access decompression are critical features that enable on-demand data access and flexible analysis workflows. However, these features can severely degrade compression quality and speed. To address these limitations, we propose a novel streaming compression framework that supports both progressive decompression and random-access decompression while maintaining high compression quality and speed. Our contributions are three-fold: (1) we design the first compression framework that simultaneously enables both progressive decompression and random-access decompression; (2) we introduce a hierarchical partitioning strategy to enable both streaming features, along with a hierarchical prediction mechanism that mitigates the impact of partitioning and achieves high compression quality -- even comparable to state-of-the-art (SOTA) non-streaming compressor SZ3; and (3) our framework delivers high compression and decompression speed, up to 6.7$\times$ faster than SZ3. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_01626 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | STZ: A High Quality and High Speed Streaming Lossy Compression Framework for Scientific Data Wang, Daoce Grosset, Pascal Pulido, Jesus Tian, Jiannan Athawale, Tushar M. Jia, Jinda Sun, Baixi Zhang, Boyuan Jin, Sian Zhao, Kai Ahrens, James Song, Fengguang Distributed, Parallel, and Cluster Computing Multimedia Error-bounded lossy compression is one of the most efficient solutions to reduce the volume of scientific data. For lossy compression, progressive decompression and random-access decompression are critical features that enable on-demand data access and flexible analysis workflows. However, these features can severely degrade compression quality and speed. To address these limitations, we propose a novel streaming compression framework that supports both progressive decompression and random-access decompression while maintaining high compression quality and speed. Our contributions are three-fold: (1) we design the first compression framework that simultaneously enables both progressive decompression and random-access decompression; (2) we introduce a hierarchical partitioning strategy to enable both streaming features, along with a hierarchical prediction mechanism that mitigates the impact of partitioning and achieves high compression quality -- even comparable to state-of-the-art (SOTA) non-streaming compressor SZ3; and (3) our framework delivers high compression and decompression speed, up to 6.7$\times$ faster than SZ3. |
| title | STZ: A High Quality and High Speed Streaming Lossy Compression Framework for Scientific Data |
| topic | Distributed, Parallel, and Cluster Computing Multimedia |
| url | https://arxiv.org/abs/2509.01626 |