Understanding Feedback Mechanisms in Machine Learning Jupyter Notebooks
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916341876785152 |
|---|---|
| author | Shome, Arumoy Cruz, Luis Spinellis, Diomidis van Deursen, Arie |
| author_facet | Shome, Arumoy Cruz, Luis Spinellis, Diomidis van Deursen, Arie |
| contents | The machine learning development lifecycle is characterized by iterative and exploratory processes that rely on feedback mechanisms to ensure data and model integrity. Despite the critical role of feedback in machine learning engineering, no prior research has been conducted to identify and understand these mechanisms. To address this knowledge gap, we mine 297.8 thousand Jupyter notebooks and analyse 2.3 million code cells. We identify three key feedback mechanisms -- assertions, print statements and last cell statements -- and further categorize them into implicit and explicit forms of feedback. Our findings reveal extensive use of implicit feedback for critical design decisions and the relatively limited adoption of explicit feedback mechanisms. By conducting detailed case studies with selected feedback instances, we uncover the potential for automated validation of critical assumptions in ML workflows using assertions. Finally, this study underscores the need for improved documentation, and provides practical recommendations on how existing feedback mechanisms in the ML development workflow can be effectively used to mitigate technical debt and enhance reproducibility. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_00153 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Understanding Feedback Mechanisms in Machine Learning Jupyter Notebooks Shome, Arumoy Cruz, Luis Spinellis, Diomidis van Deursen, Arie Software Engineering The machine learning development lifecycle is characterized by iterative and exploratory processes that rely on feedback mechanisms to ensure data and model integrity. Despite the critical role of feedback in machine learning engineering, no prior research has been conducted to identify and understand these mechanisms. To address this knowledge gap, we mine 297.8 thousand Jupyter notebooks and analyse 2.3 million code cells. We identify three key feedback mechanisms -- assertions, print statements and last cell statements -- and further categorize them into implicit and explicit forms of feedback. Our findings reveal extensive use of implicit feedback for critical design decisions and the relatively limited adoption of explicit feedback mechanisms. By conducting detailed case studies with selected feedback instances, we uncover the potential for automated validation of critical assumptions in ML workflows using assertions. Finally, this study underscores the need for improved documentation, and provides practical recommendations on how existing feedback mechanisms in the ML development workflow can be effectively used to mitigate technical debt and enhance reproducibility. |
| title | Understanding Feedback Mechanisms in Machine Learning Jupyter Notebooks |
| topic | Software Engineering |
| url | https://arxiv.org/abs/2408.00153 |