Combining Serverless and High-Performance Computing Paradigms to support ML Data-Intensive Applications
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912942968012800 |
|---|---|
| author | Staylor, Mills Sarker, Arup Kumar von Laszewski, Gregor Fox, Geoffrey Cheng, Yue Fox, Judy |
| author_facet | Staylor, Mills Sarker, Arup Kumar von Laszewski, Gregor Fox, Geoffrey Cheng, Yue Fox, Judy |
| contents | Data is found everywhere, from health and human infrastructure to the surge of sensors and the proliferation of internet-connected devices. To meet this challenge, the data engineering field has expanded significantly in recent years in both research and industry. Traditionally, data engineering, Machine Learning, and AI workloads have been run on large clusters within data center environments, requiring substantial investment in hardware and maintenance. With the rise of the public cloud, it is now possible to run large applications across nodes without owning or maintaining hardware. Serverless functions such as AWS Lambda provide horizontal scaling and precise billing without the hassle of managing traditional cloud infrastructure. However, when processing large datasets, users often rely on external storage options that are significantly slower than direct communication typical of HPC clusters. We introduce Cylon, a high-performance distributed data frame solution that has shown promising results for data processing using Python. We describe how we took inspiration from the FMI library and designed a serverless communicator to tackle communication and performance issues associated with serverless functions.
With our design, we demonstrate that the scaling efficiency of AWS Lambda achieves within 6.5% of serverful AWS (EC2) at 64 nodes, based on implementing direct communication via NAT Traversal TCP Hole Punching. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_12185 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Combining Serverless and High-Performance Computing Paradigms to support ML Data-Intensive Applications Staylor, Mills Sarker, Arup Kumar von Laszewski, Gregor Fox, Geoffrey Cheng, Yue Fox, Judy Distributed, Parallel, and Cluster Computing H.2.4, D.2.7, D.2.2 Data is found everywhere, from health and human infrastructure to the surge of sensors and the proliferation of internet-connected devices. To meet this challenge, the data engineering field has expanded significantly in recent years in both research and industry. Traditionally, data engineering, Machine Learning, and AI workloads have been run on large clusters within data center environments, requiring substantial investment in hardware and maintenance. With the rise of the public cloud, it is now possible to run large applications across nodes without owning or maintaining hardware. Serverless functions such as AWS Lambda provide horizontal scaling and precise billing without the hassle of managing traditional cloud infrastructure. However, when processing large datasets, users often rely on external storage options that are significantly slower than direct communication typical of HPC clusters. We introduce Cylon, a high-performance distributed data frame solution that has shown promising results for data processing using Python. We describe how we took inspiration from the FMI library and designed a serverless communicator to tackle communication and performance issues associated with serverless functions. With our design, we demonstrate that the scaling efficiency of AWS Lambda achieves within 6.5% of serverful AWS (EC2) at 64 nodes, based on implementing direct communication via NAT Traversal TCP Hole Punching. |
| title | Combining Serverless and High-Performance Computing Paradigms to support ML Data-Intensive Applications |
| topic | Distributed, Parallel, and Cluster Computing H.2.4, D.2.7, D.2.2 |
| url | https://arxiv.org/abs/2511.12185 |