Salvato in:
| Autori principali: | , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2508.05904 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913979949907968 |
|---|---|
| author | Baker, Brandon Brossard, Elliott Xie, Chenwei Ye, Zihao Liu, Deen Xie, Yijun Zwiegincew, Arthur Sharma, Nitya Kumar Jain, Gaurav Retunsky, Eugene Halcrow, Mike Denny-Brown, Derek Cseri, Istvan Akidau, Tyler He, Yuxiong |
| author_facet | Baker, Brandon Brossard, Elliott Xie, Chenwei Ye, Zihao Liu, Deen Xie, Yijun Zwiegincew, Arthur Sharma, Nitya Kumar Jain, Gaurav Retunsky, Eugene Halcrow, Mike Denny-Brown, Derek Cseri, Istvan Akidau, Tyler He, Yuxiong |
| contents | Snowflake revolutionized data analytics with an elastic architecture that decouples compute and storage, enabling scalable solutions supporting data architectures like data lake, data warehouse, data lakehouse, and data mesh. Building on this foundation, Snowflake has advanced its AI Data Cloud vision by introducing Snowpark, a managed turnkey solution that supports data engineering and AI and ML workloads using Python and other programming languages.
This paper outlines Snowpark's design objectives towards high performance, strong security and governance, and ease of use. We detail the architecture of Snowpark, highlighting its elastic scalability and seamless integration with Snowflake core compute infrastructure. This includes leveraging Snowflake control plane for distributed computing and employing a secure sandbox for isolating Snowflake SQL workloads from Snowpark executions. Additionally, we present core innovations in Snowpark that drive further performance enhancements, such as query initialization latency reduction through Python package caching, improved workload scheduling for customized workloads, and data skew management via efficient row redistribution. Finally, we showcase real-world case studies that illustrate Snowpark's efficiency and effectiveness for large-scale data engineering and AI and ML tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_05904 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Snowpark: Performant, Secure, User-Friendly Data Engineering and AI/ML Next To Your Data Baker, Brandon Brossard, Elliott Xie, Chenwei Ye, Zihao Liu, Deen Xie, Yijun Zwiegincew, Arthur Sharma, Nitya Kumar Jain, Gaurav Retunsky, Eugene Halcrow, Mike Denny-Brown, Derek Cseri, Istvan Akidau, Tyler He, Yuxiong Distributed, Parallel, and Cluster Computing Databases Snowflake revolutionized data analytics with an elastic architecture that decouples compute and storage, enabling scalable solutions supporting data architectures like data lake, data warehouse, data lakehouse, and data mesh. Building on this foundation, Snowflake has advanced its AI Data Cloud vision by introducing Snowpark, a managed turnkey solution that supports data engineering and AI and ML workloads using Python and other programming languages. This paper outlines Snowpark's design objectives towards high performance, strong security and governance, and ease of use. We detail the architecture of Snowpark, highlighting its elastic scalability and seamless integration with Snowflake core compute infrastructure. This includes leveraging Snowflake control plane for distributed computing and employing a secure sandbox for isolating Snowflake SQL workloads from Snowpark executions. Additionally, we present core innovations in Snowpark that drive further performance enhancements, such as query initialization latency reduction through Python package caching, improved workload scheduling for customized workloads, and data skew management via efficient row redistribution. Finally, we showcase real-world case studies that illustrate Snowpark's efficiency and effectiveness for large-scale data engineering and AI and ML tasks. |
| title | Snowpark: Performant, Secure, User-Friendly Data Engineering and AI/ML Next To Your Data |
| topic | Distributed, Parallel, and Cluster Computing Databases |
| url | https://arxiv.org/abs/2508.05904 |