XTable in Action: Seamless Interoperability in Data Lakes
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910300611018752 |
|---|---|
| author | Agrawal, Ashvin Brown, Tim Johnson, Anoop Camacho-Rodríguez, Jesús Weller, Kyle Curino, Carlo Ramakrishnan, Raghu |
| author_facet | Agrawal, Ashvin Brown, Tim Johnson, Anoop Camacho-Rodríguez, Jesús Weller, Kyle Curino, Carlo Ramakrishnan, Raghu |
| contents | Contemporary approaches to data management are increasingly relying on unified analytics and AI platforms to foster collaboration, interoperability, seamless access to reliable data, and high performance. Data Lakes featuring open standard table formats such as Delta Lake, Apache Hudi, and Apache Iceberg are central components of these data architectures. Choosing the right format for managing a table is crucial for achieving the objectives mentioned above. The challenge lies in selecting the best format, a task that is onerous and can yield temporary results, as the ideal choice may shift over time with data growth, evolving workloads, and the competitive development of table formats and processing engines. Moreover, restricting data access to a single format can hinder data sharing resulting in diminished business value over the long term. The ability to seamlessly interoperate between formats and with negligible overhead can effectively address these challenges. Our solution in this direction is an innovative omni-directional translator, XTable, that facilitates writing data in one format and reading it in any format, thus achieving the desired format interoperability. In this work, we demonstrate the effectiveness of XTable through application scenarios inspired by real-world use cases. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_09621 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | XTable in Action: Seamless Interoperability in Data Lakes Agrawal, Ashvin Brown, Tim Johnson, Anoop Camacho-Rodríguez, Jesús Weller, Kyle Curino, Carlo Ramakrishnan, Raghu Databases Contemporary approaches to data management are increasingly relying on unified analytics and AI platforms to foster collaboration, interoperability, seamless access to reliable data, and high performance. Data Lakes featuring open standard table formats such as Delta Lake, Apache Hudi, and Apache Iceberg are central components of these data architectures. Choosing the right format for managing a table is crucial for achieving the objectives mentioned above. The challenge lies in selecting the best format, a task that is onerous and can yield temporary results, as the ideal choice may shift over time with data growth, evolving workloads, and the competitive development of table formats and processing engines. Moreover, restricting data access to a single format can hinder data sharing resulting in diminished business value over the long term. The ability to seamlessly interoperate between formats and with negligible overhead can effectively address these challenges. Our solution in this direction is an innovative omni-directional translator, XTable, that facilitates writing data in one format and reading it in any format, thus achieving the desired format interoperability. In this work, we demonstrate the effectiveness of XTable through application scenarios inspired by real-world use cases. |
| title | XTable in Action: Seamless Interoperability in Data Lakes |
| topic | Databases |
| url | https://arxiv.org/abs/2401.09621 |