A Hybrid Cloud Management Plane for Data Processing Pipelines
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916685597900800 |
|---|---|
| author | Babu, Vignesh Lu, Feng Wu, Haotian Moberg, Cameron |
| author_facet | Babu, Vignesh Lu, Feng Wu, Haotian Moberg, Cameron |
| contents | As organizations increasingly rely on data-driven insights, the ability to run data intensive applications seamlessly across multiple cloud environments becomes critical for tapping into cloud innovations while complying with various security and regulatory requirements. However, big data application development and deployment remain challenging to accomplish in such environments. With the increasing containerization and modernization of big data applications, we argue that a unified control/management plane now makes sense for running these applications in hybrid cloud environments. To this end, we study the problem of building a generic hybrid-cloud management plane to radically simplify managing big data applications. A generic architecture for hybrid-cloud management, called Titchener, is proposed in this paper. Titchener comprises of independent and loosely coupled local control planes interacting with a highly available public cloud hosted global management plane. We describe a possible instantiation of Titchener based on Kubernetes and address issues related to global service discovery, network connectivity and access control enforcement. We also validate our proposed designs with a real management plane implementation based on a popular big data workflow orchestration in hybrid-cloud environments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_08225 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | A Hybrid Cloud Management Plane for Data Processing Pipelines Babu, Vignesh Lu, Feng Wu, Haotian Moberg, Cameron Distributed, Parallel, and Cluster Computing Networking and Internet Architecture As organizations increasingly rely on data-driven insights, the ability to run data intensive applications seamlessly across multiple cloud environments becomes critical for tapping into cloud innovations while complying with various security and regulatory requirements. However, big data application development and deployment remain challenging to accomplish in such environments. With the increasing containerization and modernization of big data applications, we argue that a unified control/management plane now makes sense for running these applications in hybrid cloud environments. To this end, we study the problem of building a generic hybrid-cloud management plane to radically simplify managing big data applications. A generic architecture for hybrid-cloud management, called Titchener, is proposed in this paper. Titchener comprises of independent and loosely coupled local control planes interacting with a highly available public cloud hosted global management plane. We describe a possible instantiation of Titchener based on Kubernetes and address issues related to global service discovery, network connectivity and access control enforcement. We also validate our proposed designs with a real management plane implementation based on a popular big data workflow orchestration in hybrid-cloud environments. |
| title | A Hybrid Cloud Management Plane for Data Processing Pipelines |
| topic | Distributed, Parallel, and Cluster Computing Networking and Internet Architecture |
| url | https://arxiv.org/abs/2504.08225 |