Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2411.13704 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866929599563169792 |
|---|---|
| author | Alotaibi, Rana Tian, Yuanyuan Grafberger, Stefan Camacho-Rodríguez, Jesús Bruno, Nicolas Kroth, Brian Matusevych, Sergiy Agrawal, Ashvin Behera, Mahesh Gosalia, Ashit Galindo-Legaria, Cesar Joshi, Milind Potocnik, Milan Sezgin, Beysim Li, Xiaoyu Curino, Carlo |
| author_facet | Alotaibi, Rana Tian, Yuanyuan Grafberger, Stefan Camacho-Rodríguez, Jesús Bruno, Nicolas Kroth, Brian Matusevych, Sergiy Agrawal, Ashvin Behera, Mahesh Gosalia, Ashit Galindo-Legaria, Cesar Joshi, Milind Potocnik, Milan Sezgin, Beysim Li, Xiaoyu Curino, Carlo |
| contents | Customer demand, regulatory pressure, and engineering efficiency are the driving forces behind the industry-wide trend of moving from siloed engines and services that are optimized in isolation to highly integrated solutions. This is confirmed by the wide adoption of open formats, shared component libraries, and the meteoric success of integrated data lake experiences such as Microsoft Fabric.
In this paper, we study the implications of this trend to Query Optimizer (QO) and discuss our experience of building Calcite and extending Cascades into QO components of Microsoft SQL Server, Fabric Data Warehouse (DW), and SCOPE. We weigh the pros and cons of a drastic change in direction: moving from bespoke QOs or library-sharing (à la Calcite) to rewriting the QO stack and fully embracing Query Optimizer as a Service (QOaaS). We report on some early successes and stumbles as we explore these ideas with prototypes compatible with Fabric DW and Spark. The benefits include centralized workload-level optimizations, multi-engine federation, and accelerated feature creation, but the challenges are equally daunting. We plan to engage CIDR audience in a debate on this exciting topic. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_13704 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Towards Query Optimizer as a Service (QOaaS) in a Unified LakeHouse Ecosystem: Can One QO Rule Them All? Alotaibi, Rana Tian, Yuanyuan Grafberger, Stefan Camacho-Rodríguez, Jesús Bruno, Nicolas Kroth, Brian Matusevych, Sergiy Agrawal, Ashvin Behera, Mahesh Gosalia, Ashit Galindo-Legaria, Cesar Joshi, Milind Potocnik, Milan Sezgin, Beysim Li, Xiaoyu Curino, Carlo Databases Customer demand, regulatory pressure, and engineering efficiency are the driving forces behind the industry-wide trend of moving from siloed engines and services that are optimized in isolation to highly integrated solutions. This is confirmed by the wide adoption of open formats, shared component libraries, and the meteoric success of integrated data lake experiences such as Microsoft Fabric. In this paper, we study the implications of this trend to Query Optimizer (QO) and discuss our experience of building Calcite and extending Cascades into QO components of Microsoft SQL Server, Fabric Data Warehouse (DW), and SCOPE. We weigh the pros and cons of a drastic change in direction: moving from bespoke QOs or library-sharing (à la Calcite) to rewriting the QO stack and fully embracing Query Optimizer as a Service (QOaaS). We report on some early successes and stumbles as we explore these ideas with prototypes compatible with Fabric DW and Spark. The benefits include centralized workload-level optimizations, multi-engine federation, and accelerated feature creation, but the challenges are equally daunting. We plan to engage CIDR audience in a debate on this exciting topic. |
| title | Towards Query Optimizer as a Service (QOaaS) in a Unified LakeHouse Ecosystem: Can One QO Rule Them All? |
| topic | Databases |
| url | https://arxiv.org/abs/2411.13704 |