Experiments as Code: A Concept for Reproducible, Auditable, Debuggable, Reusable, & Scalable Experiments
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2022
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915004087795712 |
|---|---|
| author | Aguilar, Leonel Gath-Morad, Michal Grübel, Jascha Ermatinger, Jasper Zhao, Hantao Wehrli, Stefan Sumner, Robert W. Zhang, Ce Helbing, Dirk Hölscher, Christoph |
| author_facet | Aguilar, Leonel Gath-Morad, Michal Grübel, Jascha Ermatinger, Jasper Zhao, Hantao Wehrli, Stefan Sumner, Robert W. Zhang, Ce Helbing, Dirk Hölscher, Christoph |
| contents | A common concern in experimental research is the auditability and reproducibility of experiments. Experiments are usually designed, provisioned, managed, and analyzed by diverse teams of specialists (e.g., researchers, technicians and engineers) and may require many resources (e.g. cloud infrastructure, specialized equipment). Even though researchers strive to document experiments accurately, this process is often lacking, making it hard to reproduce them. Moreover, when it is necessary to create a similar experiment, very often we end up "reinventing the wheel" as it is easier to start from scratch than trying to reuse existing work, thus losing valuable embedded best practices and previous experiences. In behavioral studies this has contributed to the reproducibility crisis. To tackle this challenge, we propose the "Experiments as Code" paradigm, where the whole experiment is not only documented but additionally the automation code to provision, deploy, manage, and analyze it is provided. To this end we define the Experiments as Code concept, provide a taxonomy for the components of a practical implementation, and provide a proof of concept with a simple desktop VR experiment that showcases the benefits of its "as code" representation, i.e., reproducibility, auditability, debuggability, reusability, and scalability. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2202_12050 |
| institution | arXiv |
| publishDate | 2022 |
| record_format | arxiv |
| spellingShingle | Experiments as Code: A Concept for Reproducible, Auditable, Debuggable, Reusable, & Scalable Experiments Aguilar, Leonel Gath-Morad, Michal Grübel, Jascha Ermatinger, Jasper Zhao, Hantao Wehrli, Stefan Sumner, Robert W. Zhang, Ce Helbing, Dirk Hölscher, Christoph Computers and Society K.4.3; J.4 A common concern in experimental research is the auditability and reproducibility of experiments. Experiments are usually designed, provisioned, managed, and analyzed by diverse teams of specialists (e.g., researchers, technicians and engineers) and may require many resources (e.g. cloud infrastructure, specialized equipment). Even though researchers strive to document experiments accurately, this process is often lacking, making it hard to reproduce them. Moreover, when it is necessary to create a similar experiment, very often we end up "reinventing the wheel" as it is easier to start from scratch than trying to reuse existing work, thus losing valuable embedded best practices and previous experiences. In behavioral studies this has contributed to the reproducibility crisis. To tackle this challenge, we propose the "Experiments as Code" paradigm, where the whole experiment is not only documented but additionally the automation code to provision, deploy, manage, and analyze it is provided. To this end we define the Experiments as Code concept, provide a taxonomy for the components of a practical implementation, and provide a proof of concept with a simple desktop VR experiment that showcases the benefits of its "as code" representation, i.e., reproducibility, auditability, debuggability, reusability, and scalability. |
| title | Experiments as Code: A Concept for Reproducible, Auditable, Debuggable, Reusable, & Scalable Experiments |
| topic | Computers and Society K.4.3; J.4 |
| url | https://arxiv.org/abs/2202.12050 |