Omnibenchmark: transparent, reproducible, extensible and standardized orchestration of solo and collaborative benchmarks
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911438772109312 |
|---|---|
| author | Mallona, Izaskun Luetge, Almut Carrillo, Ben Incicau, Daniel Gerber, Reto Meara, Aidan Sonrel, Anthony Soneson, Charlotte Robinson, Mark D. |
| author_facet | Mallona, Izaskun Luetge, Almut Carrillo, Ben Incicau, Daniel Gerber, Reto Meara, Aidan Sonrel, Anthony Soneson, Charlotte Robinson, Mark D. |
| contents | Benchmarking involves designing, running and disseminating rigorous performance assessments of methods, most often for data analysis and software tools, but the process can also be applied to experimental systems. Ideally, a benchmarking system is used to facilitate the benchmarking process by providing a structured entrypoint to design, coordinate, execute, and store standardized benchmarks. We describe a novel benchmarking system, Omnibenchmark, that facilitates benchmark formalization and execution in both solo and community efforts. Omnibenchmark provides a flexible benchmark plan syntax (i.e., a configuration YAML file), dynamic workflow generation based on Snakemake, S3-compatible storage handling, and reproducible software environments using environment modules, Apptainer or Conda. Such a setup provides an unprecedented flexibility such that existing benchmark designs can be forked and extended, run separately or collaboratively, giving versioned and standardized result outputs and therefore much-needed transparency to the analysis and interpretation of benchmark results. Tutorials and installation instructions are available from https://omnibenchmark.org. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_17038 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Omnibenchmark: transparent, reproducible, extensible and standardized orchestration of solo and collaborative benchmarks Mallona, Izaskun Luetge, Almut Carrillo, Ben Incicau, Daniel Gerber, Reto Meara, Aidan Sonrel, Anthony Soneson, Charlotte Robinson, Mark D. Other Quantitative Biology Benchmarking involves designing, running and disseminating rigorous performance assessments of methods, most often for data analysis and software tools, but the process can also be applied to experimental systems. Ideally, a benchmarking system is used to facilitate the benchmarking process by providing a structured entrypoint to design, coordinate, execute, and store standardized benchmarks. We describe a novel benchmarking system, Omnibenchmark, that facilitates benchmark formalization and execution in both solo and community efforts. Omnibenchmark provides a flexible benchmark plan syntax (i.e., a configuration YAML file), dynamic workflow generation based on Snakemake, S3-compatible storage handling, and reproducible software environments using environment modules, Apptainer or Conda. Such a setup provides an unprecedented flexibility such that existing benchmark designs can be forked and extended, run separately or collaboratively, giving versioned and standardized result outputs and therefore much-needed transparency to the analysis and interpretation of benchmark results. Tutorials and installation instructions are available from https://omnibenchmark.org. |
| title | Omnibenchmark: transparent, reproducible, extensible and standardized orchestration of solo and collaborative benchmarks |
| topic | Other Quantitative Biology |
| url | https://arxiv.org/abs/2409.17038 |