Accelerating Data Chunking in Deduplication Systems using Vector Instructions
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917224288092160 |
|---|---|
| author | Udayashankar, Sreeharsha Baba, Abdelrahman Al-Kiswany, Samer |
| author_facet | Udayashankar, Sreeharsha Baba, Abdelrahman Al-Kiswany, Samer |
| contents | Content-defined Chunking (CDC) algorithms dictate the overall space savings that deduplication systems achieve. However, due to their need to scan each file in its entirety, they are slow and often the main performance bottleneck within data deduplication. We present VectorCDC, a method to accelerate hashless CDC algorithms using vector CPU instructions, such as SSE / AVX. We analyzed the state-of-the-art chunking algorithms and discovered that hashless algorithms primarily use two data processing patterns to identify chunk boundaries: Extreme Byte Searches and Range Scans. VectorCDC presents a vector-friendly approach to accelerate these two patterns. Using VectorCDC, we accelerated three state-of-the-art hashless chunking algorithms: RAM, AE, and MAXP. Our evaluation shows that VectorCDC is effective on Intel, AMD, ARM, and IBM CPUs, achieving 8.35x - 26.2x higher throughput than existing vector-accelerated algorithms, and 15.3x - 207.2x higher throughput than existing unaccelerated algorithms. VectorCDC achieves this without affecting the deduplication space savings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_05797 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Accelerating Data Chunking in Deduplication Systems using Vector Instructions Udayashankar, Sreeharsha Baba, Abdelrahman Al-Kiswany, Samer Distributed, Parallel, and Cluster Computing Hardware Architecture Content-defined Chunking (CDC) algorithms dictate the overall space savings that deduplication systems achieve. However, due to their need to scan each file in its entirety, they are slow and often the main performance bottleneck within data deduplication. We present VectorCDC, a method to accelerate hashless CDC algorithms using vector CPU instructions, such as SSE / AVX. We analyzed the state-of-the-art chunking algorithms and discovered that hashless algorithms primarily use two data processing patterns to identify chunk boundaries: Extreme Byte Searches and Range Scans. VectorCDC presents a vector-friendly approach to accelerate these two patterns. Using VectorCDC, we accelerated three state-of-the-art hashless chunking algorithms: RAM, AE, and MAXP. Our evaluation shows that VectorCDC is effective on Intel, AMD, ARM, and IBM CPUs, achieving 8.35x - 26.2x higher throughput than existing vector-accelerated algorithms, and 15.3x - 207.2x higher throughput than existing unaccelerated algorithms. VectorCDC achieves this without affecting the deduplication space savings. |
| title | Accelerating Data Chunking in Deduplication Systems using Vector Instructions |
| topic | Distributed, Parallel, and Cluster Computing Hardware Architecture |
| url | https://arxiv.org/abs/2508.05797 |