Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2511.12264 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909905439424512 |
|---|---|
| author | Kononova, Anna V. van Stein, Niki Mersmann, Olaf Bäck, Thomas Bartz-Beielstein, Thomas Glasmachers, Tobias Hellwig, Michael Krey, Sebastian Kůdela, Jakub Naujoks, Boris Papenmeier, Leonard Raponi, Elena Renau, Quentin Rook, Jeroen Schäpermeier, Lennart Vermetten, Diederick Zaharie, Daniela |
| author_facet | Kononova, Anna V. van Stein, Niki Mersmann, Olaf Bäck, Thomas Bartz-Beielstein, Thomas Glasmachers, Tobias Hellwig, Michael Krey, Sebastian Kůdela, Jakub Naujoks, Boris Papenmeier, Leonard Raponi, Elena Renau, Quentin Rook, Jeroen Schäpermeier, Lennart Vermetten, Diederick Zaharie, Daniela |
| contents | Benchmarking has driven scientific progress in Evolutionary Computation, yet current practices fall short of real-world needs. Widely used synthetic suites such as BBOB and CEC isolate algorithmic phenomena but poorly reflect the structure, constraints, and information limitations of continuous and mixed-integer optimization problems in practice. This disconnect leads to the misuse of benchmarking suites for competitions, automated algorithm selection, and industrial decision-making, despite these suites being designed for different purposes.
We identify key gaps in current benchmarking practices and tooling, including limited availability of real-world-inspired problems, missing high-level features, and challenges in multi-objective and noisy settings. We propose a vision centered on curated real-world-inspired benchmarks, practitioner-accessible feature spaces and community-maintained performance databases. Real progress requires coordinated effort: A living benchmarking ecosystem that evolves with real-world insights and supports both scientific understanding and industrial use. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_12264 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Benchmarking that Matters: Rethinking Benchmarking for Practical Impact Kononova, Anna V. van Stein, Niki Mersmann, Olaf Bäck, Thomas Bartz-Beielstein, Thomas Glasmachers, Tobias Hellwig, Michael Krey, Sebastian Kůdela, Jakub Naujoks, Boris Papenmeier, Leonard Raponi, Elena Renau, Quentin Rook, Jeroen Schäpermeier, Lennart Vermetten, Diederick Zaharie, Daniela Neural and Evolutionary Computing Benchmarking has driven scientific progress in Evolutionary Computation, yet current practices fall short of real-world needs. Widely used synthetic suites such as BBOB and CEC isolate algorithmic phenomena but poorly reflect the structure, constraints, and information limitations of continuous and mixed-integer optimization problems in practice. This disconnect leads to the misuse of benchmarking suites for competitions, automated algorithm selection, and industrial decision-making, despite these suites being designed for different purposes. We identify key gaps in current benchmarking practices and tooling, including limited availability of real-world-inspired problems, missing high-level features, and challenges in multi-objective and noisy settings. We propose a vision centered on curated real-world-inspired benchmarks, practitioner-accessible feature spaces and community-maintained performance databases. Real progress requires coordinated effort: A living benchmarking ecosystem that evolves with real-world insights and supports both scientific understanding and industrial use. |
| title | Benchmarking that Matters: Rethinking Benchmarking for Practical Impact |
| topic | Neural and Evolutionary Computing |
| url | https://arxiv.org/abs/2511.12264 |