Zarr-Based Chunk-Level Cumulative Sums in Reduced Dimensions
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911010823077888 |
|---|---|
| author | Zhang, Hailiang Nguyen, Dieu My T. Smit, Christine Hegde, Mahabal |
| author_facet | Zhang, Hailiang Nguyen, Dieu My T. Smit, Christine Hegde, Mahabal |
| contents | Data analysis on massive multi-dimensional data, such as high-resolution large-region time averaging or area averaging for geospatial data, often involves calculations over a significant number of data points. While performing calculations in scalable and flexible distributed or cloud environments is a viable option, a full scan of large data volumes still serves as a computationally intensive bottleneck, leading to significant cost. This paper introduces a generic and comprehensive method to address these computational challenges. This method generates a small, size-tunable supplementary dataset that stores the cumulative sums along specific subset dimensions on top of the raw data. This minor addition unlocks rapid and cheap high-resolution large-region data analysis, making calculations over large numbers of data points feasible with small instances or even microservices in the cloud. This method is general-purpose, but is particularly well-suited for data stored in chunked, cloud-optimized formats and for services running in distributed or cloud environments. We present a Zarr extension proposal to integrate the specifications of this method and facilitate its straightforward implementation in general-purpose software applications. Benchmark tests demonstrate that this method, implemented in Amazon Web services (AWS), significantly outperforms the brute-force approach used in on-premises services. With just 5% supplemental storage, this method achieves a performance that is 3-4 orders of magnitude (~10,000 times) faster than the brute-force approach, while incurring significantly reduced computational costs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_14981 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Zarr-Based Chunk-Level Cumulative Sums in Reduced Dimensions Zhang, Hailiang Nguyen, Dieu My T. Smit, Christine Hegde, Mahabal Distributed, Parallel, and Cluster Computing Data Structures and Algorithms Data analysis on massive multi-dimensional data, such as high-resolution large-region time averaging or area averaging for geospatial data, often involves calculations over a significant number of data points. While performing calculations in scalable and flexible distributed or cloud environments is a viable option, a full scan of large data volumes still serves as a computationally intensive bottleneck, leading to significant cost. This paper introduces a generic and comprehensive method to address these computational challenges. This method generates a small, size-tunable supplementary dataset that stores the cumulative sums along specific subset dimensions on top of the raw data. This minor addition unlocks rapid and cheap high-resolution large-region data analysis, making calculations over large numbers of data points feasible with small instances or even microservices in the cloud. This method is general-purpose, but is particularly well-suited for data stored in chunked, cloud-optimized formats and for services running in distributed or cloud environments. We present a Zarr extension proposal to integrate the specifications of this method and facilitate its straightforward implementation in general-purpose software applications. Benchmark tests demonstrate that this method, implemented in Amazon Web services (AWS), significantly outperforms the brute-force approach used in on-premises services. With just 5% supplemental storage, this method achieves a performance that is 3-4 orders of magnitude (~10,000 times) faster than the brute-force approach, while incurring significantly reduced computational costs. |
| title | Zarr-Based Chunk-Level Cumulative Sums in Reduced Dimensions |
| topic | Distributed, Parallel, and Cluster Computing Data Structures and Algorithms |
| url | https://arxiv.org/abs/2506.14981 |