Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2503.20031 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915213654097920 |
|---|---|
| author | Cappello, Franck Baker, Allison Bozda, Ebru Burtscher, Martin Chard, Kyle Di, Sheng Grady, Paul Christopher O Jiang, Peng Li, Shaomeng Lindahl, Erik Lindstrom, Peter Lundborg, Magnus Zhao, Kai Liang, Xin Nagaso, Masaru Sato, Kento Singh, Amarjit Son, Seung Woo Tao, Dingwen Tian, Jiannan Underwood, Robert Yoshii, Kazutomo Lykov, Danylo Alexeev, Yuri Felker, Kyle Gerard |
| author_facet | Cappello, Franck Baker, Allison Bozda, Ebru Burtscher, Martin Chard, Kyle Di, Sheng Grady, Paul Christopher O Jiang, Peng Li, Shaomeng Lindahl, Erik Lindstrom, Peter Lundborg, Magnus Zhao, Kai Liang, Xin Nagaso, Masaru Sato, Kento Singh, Amarjit Son, Seung Woo Tao, Dingwen Tian, Jiannan Underwood, Robert Yoshii, Kazutomo Lykov, Danylo Alexeev, Yuri Felker, Kyle Gerard |
| contents | Increasing data volumes from scientific simulations and instruments (supercomputers, accelerators, telescopes) often exceed network, storage, and analysis capabilities. The scientific community's response to this challenge is scientific data reduction. Reduction can take many forms, such as triggering, sampling, filtering, quantization, and dimensionality reduction. This report focuses on a specific technique: lossy compression. Lossy compression retains all data points, leveraging correlations and controlled reduced accuracy. Quality constraints, especially for quantities of interest, are crucial for preserving scientific discoveries. User requirements also include compression ratio and speed. While many papers have been published on lossy compression techniques and reference datasets are shared by the community, there is a lack of detailed specifications of application needs that can guide lossy compression researchers and developers. This report fills this gap by reporting on the requirements and constraints of nine scientific applications covering a large spectrum of domains (climate, combustion, cosmology, fusion, light sources, molecular dynamics, quantum circuit simulation, seismology, and system logs). The report also details key lossy compression technologies (SZ, ZFP, MGARD, LC, SPERR, DCTZ, TEZip, LibPressio), discussing their history, principles, error control, hardware support, features, and impact. By presenting both application needs and compression technologies, the report aims to inspire new research to fill existing gaps. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_20031 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Lossy Compression of Scientific Data: Applications Constrains and Requirements Cappello, Franck Baker, Allison Bozda, Ebru Burtscher, Martin Chard, Kyle Di, Sheng Grady, Paul Christopher O Jiang, Peng Li, Shaomeng Lindahl, Erik Lindstrom, Peter Lundborg, Magnus Zhao, Kai Liang, Xin Nagaso, Masaru Sato, Kento Singh, Amarjit Son, Seung Woo Tao, Dingwen Tian, Jiannan Underwood, Robert Yoshii, Kazutomo Lykov, Danylo Alexeev, Yuri Felker, Kyle Gerard Instrumentation and Methods for Astrophysics Computational Engineering, Finance, and Science Increasing data volumes from scientific simulations and instruments (supercomputers, accelerators, telescopes) often exceed network, storage, and analysis capabilities. The scientific community's response to this challenge is scientific data reduction. Reduction can take many forms, such as triggering, sampling, filtering, quantization, and dimensionality reduction. This report focuses on a specific technique: lossy compression. Lossy compression retains all data points, leveraging correlations and controlled reduced accuracy. Quality constraints, especially for quantities of interest, are crucial for preserving scientific discoveries. User requirements also include compression ratio and speed. While many papers have been published on lossy compression techniques and reference datasets are shared by the community, there is a lack of detailed specifications of application needs that can guide lossy compression researchers and developers. This report fills this gap by reporting on the requirements and constraints of nine scientific applications covering a large spectrum of domains (climate, combustion, cosmology, fusion, light sources, molecular dynamics, quantum circuit simulation, seismology, and system logs). The report also details key lossy compression technologies (SZ, ZFP, MGARD, LC, SPERR, DCTZ, TEZip, LibPressio), discussing their history, principles, error control, hardware support, features, and impact. By presenting both application needs and compression technologies, the report aims to inspire new research to fill existing gaps. |
| title | Lossy Compression of Scientific Data: Applications Constrains and Requirements |
| topic | Instrumentation and Methods for Astrophysics Computational Engineering, Finance, and Science |
| url | https://arxiv.org/abs/2503.20031 |