Bauplan: zero-copy, scale-up FaaS for data pipelines
Fuente:
arXiv
Guardado en:
| Autores principales: | , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866912083192315904 |
|---|---|
| author | Tagliabue, Jacopo Caraza-Harter, Tyler Greco, Ciro |
| author_facet | Tagliabue, Jacopo Caraza-Harter, Tyler Greco, Ciro |
| contents | Chaining functions for longer workloads is a key use case for FaaS platforms in data applications. However, modern data pipelines differ significantly from typical serverless use cases (e.g., webhooks and microservices); this makes it difficult to retrofit existing pipeline frameworks due to structural constraints. In this paper, we describe these limitations in detail and introduce bauplan, a novel FaaS programming model and serverless runtime designed for data practitioners. bauplan enables users to declaratively define functional Directed Acyclic Graphs (DAGs) along with their runtime environments, which are then efficiently executed on cloud-based workers. We show that bauplan achieves both better performance and a superior developer experience for data workloads by making the trade-off of reducing generality in favor of data-awareness |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_17465 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Bauplan: zero-copy, scale-up FaaS for data pipelines Tagliabue, Jacopo Caraza-Harter, Tyler Greco, Ciro Databases Machine Learning Operating Systems Chaining functions for longer workloads is a key use case for FaaS platforms in data applications. However, modern data pipelines differ significantly from typical serverless use cases (e.g., webhooks and microservices); this makes it difficult to retrofit existing pipeline frameworks due to structural constraints. In this paper, we describe these limitations in detail and introduce bauplan, a novel FaaS programming model and serverless runtime designed for data practitioners. bauplan enables users to declaratively define functional Directed Acyclic Graphs (DAGs) along with their runtime environments, which are then efficiently executed on cloud-based workers. We show that bauplan achieves both better performance and a superior developer experience for data workloads by making the trade-off of reducing generality in favor of data-awareness |
| title | Bauplan: zero-copy, scale-up FaaS for data pipelines |
| topic | Databases Machine Learning Operating Systems |
| url | https://arxiv.org/abs/2410.17465 |