Dflow, a Python framework for constructing cloud-native AI-for-Science workflows
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909183939444736 |
|---|---|
| author | Liu, Xinzijian Han, Yanbo Li, Zhuoyuan Fan, Jiahao Zhang, Chengqian Zeng, Jinzhe Shan, Yifan Yuan, Yannan Xu, Wei-Hong Liu, Yun-Pei Zhang, Yuzhi Wen, Tongqi York, Darrin M. Zhong, Zhicheng Zheng, Hang Cheng, Jun Zhang, Linfeng Wang, Han |
| author_facet | Liu, Xinzijian Han, Yanbo Li, Zhuoyuan Fan, Jiahao Zhang, Chengqian Zeng, Jinzhe Shan, Yifan Yuan, Yannan Xu, Wei-Hong Liu, Yun-Pei Zhang, Yuzhi Wen, Tongqi York, Darrin M. Zhong, Zhicheng Zheng, Hang Cheng, Jun Zhang, Linfeng Wang, Han |
| contents | In the AI-for-science era, scientific computing scenarios such as concurrent learning and high-throughput computing demand a new generation of infrastructure that supports scalable computing resources and automated workflow management on both cloud and high-performance supercomputers. Here we introduce Dflow, an open-source Python toolkit designed for scientists to construct workflows with simple programming interfaces. It enables complex process control and task scheduling across a distributed, heterogeneous infrastructure, leveraging containers and Kubernetes for flexibility. Dflow is highly observable and can scale to thousands of concurrent nodes per workflow, enhancing the efficiency of complex scientific computing tasks. The basic unit in Dflow, known as an Operation (OP), is reusable and independent of the underlying infrastructure or context. Dozens of workflow projects have been developed based on Dflow, spanning a wide range of projects. We anticipate that the reusability of Dflow and its components will encourage more scientists to publish their workflows and OP components. These components, in turn, can be adapted and reused in various contexts, fostering greater collaboration and innovation in the scientific community. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_18392 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Dflow, a Python framework for constructing cloud-native AI-for-Science workflows Liu, Xinzijian Han, Yanbo Li, Zhuoyuan Fan, Jiahao Zhang, Chengqian Zeng, Jinzhe Shan, Yifan Yuan, Yannan Xu, Wei-Hong Liu, Yun-Pei Zhang, Yuzhi Wen, Tongqi York, Darrin M. Zhong, Zhicheng Zheng, Hang Cheng, Jun Zhang, Linfeng Wang, Han Distributed, Parallel, and Cluster Computing In the AI-for-science era, scientific computing scenarios such as concurrent learning and high-throughput computing demand a new generation of infrastructure that supports scalable computing resources and automated workflow management on both cloud and high-performance supercomputers. Here we introduce Dflow, an open-source Python toolkit designed for scientists to construct workflows with simple programming interfaces. It enables complex process control and task scheduling across a distributed, heterogeneous infrastructure, leveraging containers and Kubernetes for flexibility. Dflow is highly observable and can scale to thousands of concurrent nodes per workflow, enhancing the efficiency of complex scientific computing tasks. The basic unit in Dflow, known as an Operation (OP), is reusable and independent of the underlying infrastructure or context. Dozens of workflow projects have been developed based on Dflow, spanning a wide range of projects. We anticipate that the reusability of Dflow and its components will encourage more scientists to publish their workflows and OP components. These components, in turn, can be adapted and reused in various contexts, fostering greater collaboration and innovation in the scientific community. |
| title | Dflow, a Python framework for constructing cloud-native AI-for-Science workflows |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2404.18392 |