Inference-Sufficient Representations for High-Throughput Measurement: Lessons from Lossless Compression Benchmarks in 4D-STEM
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866918433007861760 |
|---|---|
| author | Dyck, Ondrej Lupini, Andrew R. Borisevich, Albina Chi, Miaofang Vasudevan, Rama K. Jesse, Stephen |
| author_facet | Dyck, Ondrej Lupini, Andrew R. Borisevich, Albina Chi, Miaofang Vasudevan, Rama K. Jesse, Stephen |
| contents | Four-dimensional scanning transmission electron microscopy (4D-STEM) generates multi-gigabyte datasets, creating a growing mismatch between acquisition rates and practical storage, transfer, and interactive visualization capabilities. We systematically benchmark 13 lossless compression implementations across 5 representative datasets (8~MiB to 8~GiB, 49.5--92.8\% sparsity), with 10 independent runs per method. HDF5 provides built-in gzip compression, of which gzip-9 typically achieves the highest compression ratio but is slow. We therefore evaluate widely available alternatives (via hdf5plugin), including the Blosc family. As a representative comparison, blosc\_zstd achieves compression comparable to gzip-9 (mean 13.5$\times$ vs 12.3$\times$) while compressing 19--69$\times$ faster and reading 1.9--2.6$\times$ faster across datasets. Compression ratios are deterministic, and timing measurements are highly reproducible (CV $<$2\%). Compression performance follows a power law with sparsity ($R^2 = 0.99$), ranging from 5$\times$ for moderately sparse data to 35$\times$ for highly sparse data. We identify six top-performing implementations optimized for different use cases and demonstrate that 4D-STEM data can be routinely compressed by $>$10$\times$. While these results provide practical guidance for lossless compression selection, the broader conclusion is that lossless compression preserves measurements but does not by itself guarantee sustainable high-throughput workflows. As detector rates rise, data handling will increasingly require inference-driven representations -- i.e., deciding what must be preserved to support a scientific inference, rather than defaulting to storing fully dense raw measurements. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_06221 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Inference-Sufficient Representations for High-Throughput Measurement: Lessons from Lossless Compression Benchmarks in 4D-STEM Dyck, Ondrej Lupini, Andrew R. Borisevich, Albina Chi, Miaofang Vasudevan, Rama K. Jesse, Stephen Signal Processing Materials Science Four-dimensional scanning transmission electron microscopy (4D-STEM) generates multi-gigabyte datasets, creating a growing mismatch between acquisition rates and practical storage, transfer, and interactive visualization capabilities. We systematically benchmark 13 lossless compression implementations across 5 representative datasets (8~MiB to 8~GiB, 49.5--92.8\% sparsity), with 10 independent runs per method. HDF5 provides built-in gzip compression, of which gzip-9 typically achieves the highest compression ratio but is slow. We therefore evaluate widely available alternatives (via hdf5plugin), including the Blosc family. As a representative comparison, blosc\_zstd achieves compression comparable to gzip-9 (mean 13.5$\times$ vs 12.3$\times$) while compressing 19--69$\times$ faster and reading 1.9--2.6$\times$ faster across datasets. Compression ratios are deterministic, and timing measurements are highly reproducible (CV $<$2\%). Compression performance follows a power law with sparsity ($R^2 = 0.99$), ranging from 5$\times$ for moderately sparse data to 35$\times$ for highly sparse data. We identify six top-performing implementations optimized for different use cases and demonstrate that 4D-STEM data can be routinely compressed by $>$10$\times$. While these results provide practical guidance for lossless compression selection, the broader conclusion is that lossless compression preserves measurements but does not by itself guarantee sustainable high-throughput workflows. As detector rates rise, data handling will increasingly require inference-driven representations -- i.e., deciding what must be preserved to support a scientific inference, rather than defaulting to storing fully dense raw measurements. |
| title | Inference-Sufficient Representations for High-Throughput Measurement: Lessons from Lossless Compression Benchmarks in 4D-STEM |
| topic | Signal Processing Materials Science |
| url | https://arxiv.org/abs/2604.06221 |