Towards observability of scientific applications
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913483335925760 |
|---|---|
| author | Balis, Bartosz Czerepak, Konrad Kuzma, Albert Meizner, Jan Wronski, Lukasz |
| author_facet | Balis, Bartosz Czerepak, Konrad Kuzma, Albert Meizner, Jan Wronski, Lukasz |
| contents | As software systems increase in complexity, conventional monitoring methods struggle to provide a comprehensive overview or identify performance issues, often missing unexpected problems. Observability, however, offers a holistic approach, providing methods and tools that gather and analyze detailed telemetry data to uncover hidden issues. Originally developed for cloud-native systems, modern observability is less prevalent in scientific computing, particularly in HPC clusters, due to differences in application architecture, execution environments, and technology stacks. This paper proposes and evaluates an end-to-end observability solution tailored for scientific computing in HPC environments. We address several challenges, including collection of application-level metrics, instrumentation, context propagation, and tracing. We argue that typical dashboards with charts are not sufficient for advanced observability-driven analysis of scientific applications. Consequently, we propose a different approach based on data analysis using DataFrames and a Jupyter environment. The proposed solution is implemented and evaluated on two medical scientific pipelines running on an HPC cluster. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_15439 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Towards observability of scientific applications Balis, Bartosz Czerepak, Konrad Kuzma, Albert Meizner, Jan Wronski, Lukasz Distributed, Parallel, and Cluster Computing As software systems increase in complexity, conventional monitoring methods struggle to provide a comprehensive overview or identify performance issues, often missing unexpected problems. Observability, however, offers a holistic approach, providing methods and tools that gather and analyze detailed telemetry data to uncover hidden issues. Originally developed for cloud-native systems, modern observability is less prevalent in scientific computing, particularly in HPC clusters, due to differences in application architecture, execution environments, and technology stacks. This paper proposes and evaluates an end-to-end observability solution tailored for scientific computing in HPC environments. We address several challenges, including collection of application-level metrics, instrumentation, context propagation, and tracing. We argue that typical dashboards with charts are not sufficient for advanced observability-driven analysis of scientific applications. Consequently, we propose a different approach based on data analysis using DataFrames and a Jupyter environment. The proposed solution is implemented and evaluated on two medical scientific pipelines running on an HPC cluster. |
| title | Towards observability of scientific applications |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2408.15439 |