Reducing the Impact of I/O Contention in Numerical Weather Prediction Workflows at Scale Using DAOS
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911827562070016 |
|---|---|
| author | Manubens, Nicolau Smart, Simon D. Danovaro, Emanuele Quintino, Tiago Jackson, Adrian |
| author_facet | Manubens, Nicolau Smart, Simon D. Danovaro, Emanuele Quintino, Tiago Jackson, Adrian |
| contents | Operational Numerical Weather Prediction (NWP) workflows are highly data-intensive. Data volumes have increased by many orders of magnitude over the last 40 years, and are expected to continue to do so, especially given the upcoming adoption of Machine Learning in forecast processes. Parallel POSIX-compliant file systems have been the dominant paradigm in data storage and exchange in HPC workflows for many years. This paper presents ECMWF's move beyond the POSIX paradigm, implementing a backend for their storage library to support DAOS -- a novel high-performance object store designed for massively distributed Non-Volatile Memory. This system is demonstrated to be able to outperform the highly mature and optimised POSIX backend when used under high load and contention, as per typical forecast workflow I/O patterns. This work constitutes a significant step forward, beyond the performance constraints imposed by POSIX semantics. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_03107 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Reducing the Impact of I/O Contention in Numerical Weather Prediction Workflows at Scale Using DAOS Manubens, Nicolau Smart, Simon D. Danovaro, Emanuele Quintino, Tiago Jackson, Adrian Distributed, Parallel, and Cluster Computing Operational Numerical Weather Prediction (NWP) workflows are highly data-intensive. Data volumes have increased by many orders of magnitude over the last 40 years, and are expected to continue to do so, especially given the upcoming adoption of Machine Learning in forecast processes. Parallel POSIX-compliant file systems have been the dominant paradigm in data storage and exchange in HPC workflows for many years. This paper presents ECMWF's move beyond the POSIX paradigm, implementing a backend for their storage library to support DAOS -- a novel high-performance object store designed for massively distributed Non-Volatile Memory. This system is demonstrated to be able to outperform the highly mature and optimised POSIX backend when used under high load and contention, as per typical forecast workflow I/O patterns. This work constitutes a significant step forward, beyond the performance constraints imposed by POSIX semantics. |
| title | Reducing the Impact of I/O Contention in Numerical Weather Prediction Workflows at Scale Using DAOS |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2404.03107 |