Data Caching for Enterprise-Grade Petabyte-Scale OLAP
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910479274737664 |
|---|---|
| author | Tang, Chunxu Fan, Bin Zhao, Jing Liang, Chen Wang, Yi Wang, Beinan Qiu, Ziyue Qiu, Lu Ding, Bowen Sun, Shouzhuo Che, Saiguang Mai, Jiaming Chen, Shouwei Zhu, Yu Xie, Jianjian Yutian Sun Li, Yao Zhang, Yangjun Wang, Ke Chen, Mingmin |
| author_facet | Tang, Chunxu Fan, Bin Zhao, Jing Liang, Chen Wang, Yi Wang, Beinan Qiu, Ziyue Qiu, Lu Ding, Bowen Sun, Shouzhuo Che, Saiguang Mai, Jiaming Chen, Shouwei Zhu, Yu Xie, Jianjian Yutian Sun Li, Yao Zhang, Yangjun Wang, Ke Chen, Mingmin |
| contents | With the exponential growth of data and evolving use cases, petabyte-scale OLAP data platforms are increasingly adopting a model that decouples compute from storage. This shift, evident in organizations like Uber and Meta, introduces operational challenges including massive, read-heavy I/O traffic with potential throttling, as well as skewed and fragmented data access patterns. Addressing these challenges, this paper introduces the Alluxio local (edge) cache, a highly effective architectural optimization tailored for such environments. This embeddable cache, optimized for petabyte-scale data analytics, leverages local SSD resources to alleviate network I/O and API call pressures, significantly improving data transfer efficiency. Integrated with OLAP systems like Presto and storage services like HDFS, the Alluxio local cache has demonstrated its effectiveness in handling large-scale, enterprise-grade workloads over three years of deployment at Uber and Meta. We share insights and operational experiences in implementing these optimizations, providing valuable perspectives on managing modern, massive-scale OLAP workloads. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_05962 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Data Caching for Enterprise-Grade Petabyte-Scale OLAP Tang, Chunxu Fan, Bin Zhao, Jing Liang, Chen Wang, Yi Wang, Beinan Qiu, Ziyue Qiu, Lu Ding, Bowen Sun, Shouzhuo Che, Saiguang Mai, Jiaming Chen, Shouwei Zhu, Yu Xie, Jianjian Yutian Sun Li, Yao Zhang, Yangjun Wang, Ke Chen, Mingmin Distributed, Parallel, and Cluster Computing Databases With the exponential growth of data and evolving use cases, petabyte-scale OLAP data platforms are increasingly adopting a model that decouples compute from storage. This shift, evident in organizations like Uber and Meta, introduces operational challenges including massive, read-heavy I/O traffic with potential throttling, as well as skewed and fragmented data access patterns. Addressing these challenges, this paper introduces the Alluxio local (edge) cache, a highly effective architectural optimization tailored for such environments. This embeddable cache, optimized for petabyte-scale data analytics, leverages local SSD resources to alleviate network I/O and API call pressures, significantly improving data transfer efficiency. Integrated with OLAP systems like Presto and storage services like HDFS, the Alluxio local cache has demonstrated its effectiveness in handling large-scale, enterprise-grade workloads over three years of deployment at Uber and Meta. We share insights and operational experiences in implementing these optimizations, providing valuable perspectives on managing modern, massive-scale OLAP workloads. |
| title | Data Caching for Enterprise-Grade Petabyte-Scale OLAP |
| topic | Distributed, Parallel, and Cluster Computing Databases |
| url | https://arxiv.org/abs/2406.05962 |