Cover Image

iDDS: Intelligent Distributed Dispatch and Scheduling for Workflow Orchestration

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Guan, Wen, Maeno, Tadashi, Alekseev, Aleksandr, Megino, Fernando Harald Barreiro, De, Kaushik, Karavakis, Edward, Klimentov, Alexei, Korchuganova, Tatiana, Lin, FaHui, Nilsson, Paul, Wenaus, Torre, Yang, Zhaoyu, Zhao, Xin
Format:	Preprint
Published:	2025
Subjects:	Distributed, Parallel, and Cluster Computing
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

_version_	1866908724290912256
author	Guan, Wen Maeno, Tadashi Alekseev, Aleksandr Megino, Fernando Harald Barreiro De, Kaushik Karavakis, Edward Klimentov, Alexei Korchuganova, Tatiana Lin, FaHui Nilsson, Paul Wenaus, Torre Yang, Zhaoyu Zhao, Xin
author_facet	Guan, Wen Maeno, Tadashi Alekseev, Aleksandr Megino, Fernando Harald Barreiro De, Kaushik Karavakis, Edward Klimentov, Alexei Korchuganova, Tatiana Lin, FaHui Nilsson, Paul Wenaus, Torre Yang, Zhaoyu Zhao, Xin
contents	The intelligent Distributed Dispatch and Scheduling (iDDS) service is a versatile workflow orchestration system designed for large-scale, distributed scientific computing. iDDS extends traditional workload and data management by integrating data-aware execution, conditional logic, and programmable workflows, enabling automation of complex and dynamic processing pipelines. Originally developed for the ATLAS experiment at the Large Hadron Collider, iDDS has evolved into an experiment-agnostic platform that supports both template-driven workflows and a Function-as-a-Task model for Python-based orchestration. This paper presents the architecture and core components of iDDS, highlighting its scalability, modular message-driven design, and integration with systems such as PanDA and Rucio. We demonstrate its versatility through real-world use cases: fine-grained tape resource optimization for ATLAS, orchestration of large Directed Acyclic Graph (DAG) workflows for the Rubin Observatory, distributed hyperparameter optimization for machine learning applications, active learning for physics analyses, and AI-assisted detector design at the Electron-Ion Collider. By unifying workload scheduling, data movement, and adaptive decision-making, iDDS reduces operational overhead and enables reproducible, high-throughput workflows across heterogeneous infrastructures. We conclude with current challenges and future directions, including interactive, cloud-native, and serverless workflow support.
format	Preprint
id	arxiv_https___arxiv_org_abs_2510_02930
institution	arXiv
publishDate	2025
record_format	arxiv
spellingShingle	iDDS: Intelligent Distributed Dispatch and Scheduling for Workflow Orchestration Guan, Wen Maeno, Tadashi Alekseev, Aleksandr Megino, Fernando Harald Barreiro De, Kaushik Karavakis, Edward Klimentov, Alexei Korchuganova, Tatiana Lin, FaHui Nilsson, Paul Wenaus, Torre Yang, Zhaoyu Zhao, Xin Distributed, Parallel, and Cluster Computing The intelligent Distributed Dispatch and Scheduling (iDDS) service is a versatile workflow orchestration system designed for large-scale, distributed scientific computing. iDDS extends traditional workload and data management by integrating data-aware execution, conditional logic, and programmable workflows, enabling automation of complex and dynamic processing pipelines. Originally developed for the ATLAS experiment at the Large Hadron Collider, iDDS has evolved into an experiment-agnostic platform that supports both template-driven workflows and a Function-as-a-Task model for Python-based orchestration. This paper presents the architecture and core components of iDDS, highlighting its scalability, modular message-driven design, and integration with systems such as PanDA and Rucio. We demonstrate its versatility through real-world use cases: fine-grained tape resource optimization for ATLAS, orchestration of large Directed Acyclic Graph (DAG) workflows for the Rubin Observatory, distributed hyperparameter optimization for machine learning applications, active learning for physics analyses, and AI-assisted detector design at the Electron-Ion Collider. By unifying workload scheduling, data movement, and adaptive decision-making, iDDS reduces operational overhead and enables reproducible, high-throughput workflows across heterogeneous infrastructures. We conclude with current challenges and future directions, including interactive, cloud-native, and serverless workflow support.
title	iDDS: Intelligent Distributed Dispatch and Scheduling for Workflow Orchestration
topic	Distributed, Parallel, and Cluster Computing
url	https://arxiv.org/abs/2510.02930