iDDS: Intelligent Distributed Dispatch and Scheduling for Workflow Orchestration
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866908724290912256 |
|---|---|
| author | Guan, Wen Maeno, Tadashi Alekseev, Aleksandr Megino, Fernando Harald Barreiro De, Kaushik Karavakis, Edward Klimentov, Alexei Korchuganova, Tatiana Lin, FaHui Nilsson, Paul Wenaus, Torre Yang, Zhaoyu Zhao, Xin |
| author_facet | Guan, Wen Maeno, Tadashi Alekseev, Aleksandr Megino, Fernando Harald Barreiro De, Kaushik Karavakis, Edward Klimentov, Alexei Korchuganova, Tatiana Lin, FaHui Nilsson, Paul Wenaus, Torre Yang, Zhaoyu Zhao, Xin |
| contents | The intelligent Distributed Dispatch and Scheduling (iDDS) service is a versatile workflow orchestration system designed for large-scale, distributed scientific computing. iDDS extends traditional workload and data management by integrating data-aware execution, conditional logic, and programmable workflows, enabling automation of complex and dynamic processing pipelines. Originally developed for the ATLAS experiment at the Large Hadron Collider, iDDS has evolved into an experiment-agnostic platform that supports both template-driven workflows and a Function-as-a-Task model for Python-based orchestration.
This paper presents the architecture and core components of iDDS, highlighting its scalability, modular message-driven design, and integration with systems such as PanDA and Rucio. We demonstrate its versatility through real-world use cases: fine-grained tape resource optimization for ATLAS, orchestration of large Directed Acyclic Graph (DAG) workflows for the Rubin Observatory, distributed hyperparameter optimization for machine learning applications, active learning for physics analyses, and AI-assisted detector design at the Electron-Ion Collider.
By unifying workload scheduling, data movement, and adaptive decision-making, iDDS reduces operational overhead and enables reproducible, high-throughput workflows across heterogeneous infrastructures. We conclude with current challenges and future directions, including interactive, cloud-native, and serverless workflow support. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_02930 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | iDDS: Intelligent Distributed Dispatch and Scheduling for Workflow Orchestration Guan, Wen Maeno, Tadashi Alekseev, Aleksandr Megino, Fernando Harald Barreiro De, Kaushik Karavakis, Edward Klimentov, Alexei Korchuganova, Tatiana Lin, FaHui Nilsson, Paul Wenaus, Torre Yang, Zhaoyu Zhao, Xin Distributed, Parallel, and Cluster Computing The intelligent Distributed Dispatch and Scheduling (iDDS) service is a versatile workflow orchestration system designed for large-scale, distributed scientific computing. iDDS extends traditional workload and data management by integrating data-aware execution, conditional logic, and programmable workflows, enabling automation of complex and dynamic processing pipelines. Originally developed for the ATLAS experiment at the Large Hadron Collider, iDDS has evolved into an experiment-agnostic platform that supports both template-driven workflows and a Function-as-a-Task model for Python-based orchestration. This paper presents the architecture and core components of iDDS, highlighting its scalability, modular message-driven design, and integration with systems such as PanDA and Rucio. We demonstrate its versatility through real-world use cases: fine-grained tape resource optimization for ATLAS, orchestration of large Directed Acyclic Graph (DAG) workflows for the Rubin Observatory, distributed hyperparameter optimization for machine learning applications, active learning for physics analyses, and AI-assisted detector design at the Electron-Ion Collider. By unifying workload scheduling, data movement, and adaptive decision-making, iDDS reduces operational overhead and enables reproducible, high-throughput workflows across heterogeneous infrastructures. We conclude with current challenges and future directions, including interactive, cloud-native, and serverless workflow support. |
| title | iDDS: Intelligent Distributed Dispatch and Scheduling for Workflow Orchestration |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2510.02930 |