Employing Artificial Intelligence to Steer Exascale Workflows with Colmena
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913480989212672 |
|---|---|
| author | Ward, Logan Pauloski, J. Gregory Hayot-Sasson, Valerie Babuji, Yadu Brace, Alexander Chard, Ryan Chard, Kyle Thakur, Rajeev Foster, Ian |
| author_facet | Ward, Logan Pauloski, J. Gregory Hayot-Sasson, Valerie Babuji, Yadu Brace, Alexander Chard, Ryan Chard, Kyle Thakur, Rajeev Foster, Ian |
| contents | Computational workflows are a common class of application on supercomputers, yet the loosely coupled and heterogeneous nature of workflows often fails to take full advantage of their capabilities. We created Colmena to leverage the massive parallelism of a supercomputer by using Artificial Intelligence (AI) to learn from and adapt a workflow as it executes. Colmena allows scientists to define how their application should respond to events (e.g., task completion) as a series of cooperative agents. In this paper, we describe the design of Colmena, the challenges we overcame while deploying applications on exascale systems, and the science workflows we have enhanced through interweaving AI. The scaling challenges we discuss include developing steering strategies that maximize node utilization, introducing data fabrics that reduce communication overhead of data-intensive tasks, and implementing workflow tasks that cache costly operations between invocations. These innovations coupled with a variety of application patterns accessible through our agent-based steering model have enabled science advances in chemistry, biophysics, and materials science using different types of AI. Our vision is that Colmena will spur creative solutions that harness AI across many domains of scientific computing. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_14434 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Employing Artificial Intelligence to Steer Exascale Workflows with Colmena Ward, Logan Pauloski, J. Gregory Hayot-Sasson, Valerie Babuji, Yadu Brace, Alexander Chard, Ryan Chard, Kyle Thakur, Rajeev Foster, Ian Distributed, Parallel, and Cluster Computing Machine Learning Computational workflows are a common class of application on supercomputers, yet the loosely coupled and heterogeneous nature of workflows often fails to take full advantage of their capabilities. We created Colmena to leverage the massive parallelism of a supercomputer by using Artificial Intelligence (AI) to learn from and adapt a workflow as it executes. Colmena allows scientists to define how their application should respond to events (e.g., task completion) as a series of cooperative agents. In this paper, we describe the design of Colmena, the challenges we overcame while deploying applications on exascale systems, and the science workflows we have enhanced through interweaving AI. The scaling challenges we discuss include developing steering strategies that maximize node utilization, introducing data fabrics that reduce communication overhead of data-intensive tasks, and implementing workflow tasks that cache costly operations between invocations. These innovations coupled with a variety of application patterns accessible through our agent-based steering model have enabled science advances in chemistry, biophysics, and materials science using different types of AI. Our vision is that Colmena will spur creative solutions that harness AI across many domains of scientific computing. |
| title | Employing Artificial Intelligence to Steer Exascale Workflows with Colmena |
| topic | Distributed, Parallel, and Cluster Computing Machine Learning |
| url | https://arxiv.org/abs/2408.14434 |