The Chonkers Algorithm: Content-Defined Chunking with Provable Strict Guarantees on Size and Locality
Fuente:
arXiv
Saved in:
| Main Author: | |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911167734087680 |
|---|---|
| author | Berger, Benjamin |
| author_facet | Berger, Benjamin |
| contents | This paper presents the Chonkers algorithm, a novel content-defined chunking method providing simultaneous provable strict guarantees on chunk size and edit locality. Unlike existing algorithms such as Rabin fingerprinting and anchor-based methods, Chonkers achieves bounded propagation of edits and precise control over chunk sizes. I describe the algorithm's layered structure that allows for combination with other chunking algorithms, the theoretical guarantees it provides, implementation considerations, and introduce the Yarn datatype, a deduplicated, merge-tree-based string representation benefiting from Chonkers' strict guarantees. Finally, I experimentally compare Chonkers' ability to deduplicate versioned data to other algorithms and evaluate Chonkers on three corpora with respect to the actually occurring chunk sizes and edit locality, and find that it performs much better in practice than the proved guarantees. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_11121 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | The Chonkers Algorithm: Content-Defined Chunking with Provable Strict Guarantees on Size and Locality Berger, Benjamin Data Structures and Algorithms F.2.2; E.1; H.3.1 This paper presents the Chonkers algorithm, a novel content-defined chunking method providing simultaneous provable strict guarantees on chunk size and edit locality. Unlike existing algorithms such as Rabin fingerprinting and anchor-based methods, Chonkers achieves bounded propagation of edits and precise control over chunk sizes. I describe the algorithm's layered structure that allows for combination with other chunking algorithms, the theoretical guarantees it provides, implementation considerations, and introduce the Yarn datatype, a deduplicated, merge-tree-based string representation benefiting from Chonkers' strict guarantees. Finally, I experimentally compare Chonkers' ability to deduplicate versioned data to other algorithms and evaluate Chonkers on three corpora with respect to the actually occurring chunk sizes and edit locality, and find that it performs much better in practice than the proved guarantees. |
| title | The Chonkers Algorithm: Content-Defined Chunking with Provable Strict Guarantees on Size and Locality |
| topic | Data Structures and Algorithms F.2.2; E.1; H.3.1 |
| url | https://arxiv.org/abs/2509.11121 |