Zerrow: True Zero-Copy Arrow Pipelines in Bauplan
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866918019391815680 |
|---|---|
| author | Dai, Yifan Tagliabue, Jacopo Arpaci-Dusseau, Andrea Arpaci-Dusseau, Remzi Caraza-Harter, Tyler R. |
| author_facet | Dai, Yifan Tagliabue, Jacopo Arpaci-Dusseau, Andrea Arpaci-Dusseau, Remzi Caraza-Harter, Tyler R. |
| contents | Bauplan is a FaaS-based lakehouse specifically built for data pipelines: its execution engine uses Apache Arrow for data passing between the nodes in the DAG. While Arrow is known as the "zero copy format", in practice, limited Linux kernel support for shared memory makes it difficult to avoid copying entirely. In this work, we introduce several new techniques to eliminate nearly all copying from pipelines: in particular, we implement a new kernel module that performs de-anonymization, thus eliminating a copy to intermediate data. We conclude by sharing our preliminary evaluation on different workloads types, as well as discussing our plan for future improvements. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_06151 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Zerrow: True Zero-Copy Arrow Pipelines in Bauplan Dai, Yifan Tagliabue, Jacopo Arpaci-Dusseau, Andrea Arpaci-Dusseau, Remzi Caraza-Harter, Tyler R. Operating Systems Databases Bauplan is a FaaS-based lakehouse specifically built for data pipelines: its execution engine uses Apache Arrow for data passing between the nodes in the DAG. While Arrow is known as the "zero copy format", in practice, limited Linux kernel support for shared memory makes it difficult to avoid copying entirely. In this work, we introduce several new techniques to eliminate nearly all copying from pipelines: in particular, we implement a new kernel module that performs de-anonymization, thus eliminating a copy to intermediate data. We conclude by sharing our preliminary evaluation on different workloads types, as well as discussing our plan for future improvements. |
| title | Zerrow: True Zero-Copy Arrow Pipelines in Bauplan |
| topic | Operating Systems Databases |
| url | https://arxiv.org/abs/2504.06151 |