Saved in:
| Main Author: | |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2512.06852 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911306364223488 |
|---|---|
| author | Chinthareddy, Manideep Reddy |
| author_facet | Chinthareddy, Manideep Reddy |
| contents | Many managed key-value and NoSQL databases - such as Amazon DynamoDB, Azure Cosmos DB, and Google Cloud Firestore - enforce strict maximum item sizes (e.g., 400 KB in DynamoDB). This constraint imposes significant architectural challenges for applications requiring low-latency, multi-region access to objects that exceed these limits. The standard industry recommendation is to offload payloads to object storage (e.g., Amazon S3) while retaining a pointer in the database. While cost-efficient, this "pointer pattern" introduces network overhead and exposes applications to non-deterministic replication lag between the database and the object store, creating race conditions in active-active architectures.
This paper presents a "chunked-object" pattern that persists large logical entities as sets of ordered chunks within the database itself. We precisely define the pattern and provide a reference implementation using Amazon DynamoDB Global Tables. The design generalizes to any key-value store with per-item size limits and multi-region replication. We evaluate the approach using telemetry from a production system processing over 200,000 transactions per hour. Results demonstrate that the chunked-object pattern eliminates cross-system replication lag hazards and reduces p99 cross-region time-to-consistency for 1 MB payloads by keeping data and metadata within a single consistency domain. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_06852 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | A Chunked-Object Pattern for Multi-Region Large Payload Storage in Managed NoSQL Databases Chinthareddy, Manideep Reddy Databases Distributed, Parallel, and Cluster Computing Many managed key-value and NoSQL databases - such as Amazon DynamoDB, Azure Cosmos DB, and Google Cloud Firestore - enforce strict maximum item sizes (e.g., 400 KB in DynamoDB). This constraint imposes significant architectural challenges for applications requiring low-latency, multi-region access to objects that exceed these limits. The standard industry recommendation is to offload payloads to object storage (e.g., Amazon S3) while retaining a pointer in the database. While cost-efficient, this "pointer pattern" introduces network overhead and exposes applications to non-deterministic replication lag between the database and the object store, creating race conditions in active-active architectures. This paper presents a "chunked-object" pattern that persists large logical entities as sets of ordered chunks within the database itself. We precisely define the pattern and provide a reference implementation using Amazon DynamoDB Global Tables. The design generalizes to any key-value store with per-item size limits and multi-region replication. We evaluate the approach using telemetry from a production system processing over 200,000 transactions per hour. Results demonstrate that the chunked-object pattern eliminates cross-system replication lag hazards and reduces p99 cross-region time-to-consistency for 1 MB payloads by keeping data and metadata within a single consistency domain. |
| title | A Chunked-Object Pattern for Multi-Region Large Payload Storage in Managed NoSQL Databases |
| topic | Databases Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2512.06852 |