Leveraging Apache Arrow for Zero-copy, Zero-serialization Cluster Shared Memory
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911827435192320 |
|---|---|
| author | Groet, Philip Hoozemans, Joost Grapentin, Andreas Eberhardt, Felix Al-Ars, Zaid Hofstee, H. Peter |
| author_facet | Groet, Philip Hoozemans, Joost Grapentin, Andreas Eberhardt, Felix Al-Ars, Zaid Hofstee, H. Peter |
| contents | This paper describes a distributed implementation of Apache Arrow that can leverage cluster-shared load-store addressable memory that is hardware-coherent only within each node. The implementation is built on the ThymesisFlow prototype that leverages the OpenCAPI interface to create a shared address space across a cluster. While Apache Arrow structures are immutable, simplifying their use in a cluster shared memory, this paper creates distributed Apache Arrow tables and makes them accessible in each node. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_03030 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Leveraging Apache Arrow for Zero-copy, Zero-serialization Cluster Shared Memory Groet, Philip Hoozemans, Joost Grapentin, Andreas Eberhardt, Felix Al-Ars, Zaid Hofstee, H. Peter Emerging Technologies This paper describes a distributed implementation of Apache Arrow that can leverage cluster-shared load-store addressable memory that is hardware-coherent only within each node. The implementation is built on the ThymesisFlow prototype that leverages the OpenCAPI interface to create a shared address space across a cluster. While Apache Arrow structures are immutable, simplifying their use in a cluster shared memory, this paper creates distributed Apache Arrow tables and makes them accessible in each node. |
| title | Leveraging Apache Arrow for Zero-copy, Zero-serialization Cluster Shared Memory |
| topic | Emerging Technologies |
| url | https://arxiv.org/abs/2404.03030 |