Union: An Automatic Workload Manager for Accelerating Network Simulation
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913298287427584 |
|---|---|
| author | Wang, Xin Mubarak, Misbah Kang, Yao Ross, Robert B. Lan, Zhiling |
| author_facet | Wang, Xin Mubarak, Misbah Kang, Yao Ross, Robert B. Lan, Zhiling |
| contents | With the rapid growth of the machine learning applications, the workloads of future HPC systems are anticipated to be a mix of scientific simulation, big data analytics, and machine learning applications. Simulation is a great research vehicle to understand the performance implications of co-running scientific applications with big data and machine learning workloads on large-scale systems. In this paper, we present Union, a workload manager that provides an automatic framework to facilitate hybrid workload simulation in CODES. Furthermore, we use Union, along with CODES, to investigate various hybrid workloads composed of traditional simulation applications and emerging learning applications on two dragonfly systems. The experiment results show that both message latency and communication time are important performance metrics to evaluate network interference. Network interference on HPC applications is more reflected by the message latency variation, whereas ML application performance depends more on the communication time. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_17036 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Union: An Automatic Workload Manager for Accelerating Network Simulation Wang, Xin Mubarak, Misbah Kang, Yao Ross, Robert B. Lan, Zhiling Distributed, Parallel, and Cluster Computing With the rapid growth of the machine learning applications, the workloads of future HPC systems are anticipated to be a mix of scientific simulation, big data analytics, and machine learning applications. Simulation is a great research vehicle to understand the performance implications of co-running scientific applications with big data and machine learning workloads on large-scale systems. In this paper, we present Union, a workload manager that provides an automatic framework to facilitate hybrid workload simulation in CODES. Furthermore, we use Union, along with CODES, to investigate various hybrid workloads composed of traditional simulation applications and emerging learning applications on two dragonfly systems. The experiment results show that both message latency and communication time are important performance metrics to evaluate network interference. Network interference on HPC applications is more reflected by the message latency variation, whereas ML application performance depends more on the communication time. |
| title | Union: An Automatic Workload Manager for Accelerating Network Simulation |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2403.17036 |