Croissant Tasks: A Metadata Format for Reproducible Machine Learning Evaluations
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913170124177408 |
|---|---|
| author | Benjelloun, Omar Bianco, Leonardo Martins Guyon, Isabelle Khuong, Thanh Gia Hieu Lebensold, Jonathan Lobentanzer, Sebastian Oala, Luis Rachmat, Benedictus Kent Ullah, Ihsan Vahidi, Peyman Vanschoren, Joaquin |
| author_facet | Benjelloun, Omar Bianco, Leonardo Martins Guyon, Isabelle Khuong, Thanh Gia Hieu Lebensold, Jonathan Lobentanzer, Sebastian Oala, Luis Rachmat, Benedictus Kent Ullah, Ihsan Vahidi, Peyman Vanschoren, Joaquin |
| contents | Reproducibility is fundamental to the scientific method, yet remains a critical challenge in machine learning. Contributing factors include underspecified execution details and brittle software environments. Human-centric remedies, such as checklists and manual verification, help but require intensive effort and fail to scale. To address this, we introduce Croissant Tasks: a declarative, machine-actionable metadata format that abstracts low-level implementation details into high-level specifications. This format enables conceptual reproducibility: verifying claims via independent, agent-generated implementations rather than brittle source code replication. We contribute: (1) the Croissant Tasks specification, formally decoupling task problem from solution; (2) an automated LLM pipeline that retrofits existing benchmarks into this format; and (3) empirical validation showing autonomous agents can ingest these specifications to generate functional, accurate reproduction pipelines from scratch. We envision this format as a new foundation for automated and conceptual reproducibility in machine learning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_29786 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Croissant Tasks: A Metadata Format for Reproducible Machine Learning Evaluations Benjelloun, Omar Bianco, Leonardo Martins Guyon, Isabelle Khuong, Thanh Gia Hieu Lebensold, Jonathan Lobentanzer, Sebastian Oala, Luis Rachmat, Benedictus Kent Ullah, Ihsan Vahidi, Peyman Vanschoren, Joaquin Artificial Intelligence Reproducibility is fundamental to the scientific method, yet remains a critical challenge in machine learning. Contributing factors include underspecified execution details and brittle software environments. Human-centric remedies, such as checklists and manual verification, help but require intensive effort and fail to scale. To address this, we introduce Croissant Tasks: a declarative, machine-actionable metadata format that abstracts low-level implementation details into high-level specifications. This format enables conceptual reproducibility: verifying claims via independent, agent-generated implementations rather than brittle source code replication. We contribute: (1) the Croissant Tasks specification, formally decoupling task problem from solution; (2) an automated LLM pipeline that retrofits existing benchmarks into this format; and (3) empirical validation showing autonomous agents can ingest these specifications to generate functional, accurate reproduction pipelines from scratch. We envision this format as a new foundation for automated and conceptual reproducibility in machine learning. |
| title | Croissant Tasks: A Metadata Format for Reproducible Machine Learning Evaluations |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2605.29786 |