Statistical Multicriteria Benchmarking via the GSD-Front
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917686517170176 |
|---|---|
| author | Jansen, Christoph Schollmeyer, Georg Rodemann, Julian Blocher, Hannah Augustin, Thomas |
| author_facet | Jansen, Christoph Schollmeyer, Georg Rodemann, Julian Blocher, Hannah Augustin, Thomas |
| contents | Given the vast number of classifiers that have been (and continue to be) proposed, reliable methods for comparing them are becoming increasingly important. The desire for reliability is broken down into three main aspects: (1) Comparisons should allow for different quality metrics simultaneously. (2) Comparisons should take into account the statistical uncertainty induced by the choice of benchmark suite. (3) The robustness of the comparisons under small deviations in the underlying assumptions should be verifiable. To address (1), we propose to compare classifiers using a generalized stochastic dominance ordering (GSD) and present the GSD-front as an information-efficient alternative to the classical Pareto-front. For (2), we propose a consistent statistical estimator for the GSD-front and construct a statistical test for whether a (potentially new) classifier lies in the GSD-front of a set of state-of-the-art classifiers. For (3), we relax our proposed test using techniques from robust statistics and imprecise probabilities. We illustrate our concepts on the benchmark suite PMLB and on the platform OpenML. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_03924 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Statistical Multicriteria Benchmarking via the GSD-Front Jansen, Christoph Schollmeyer, Georg Rodemann, Julian Blocher, Hannah Augustin, Thomas Machine Learning Methodology 62G05, 62G35, 62G09, 62G10 G.3 Given the vast number of classifiers that have been (and continue to be) proposed, reliable methods for comparing them are becoming increasingly important. The desire for reliability is broken down into three main aspects: (1) Comparisons should allow for different quality metrics simultaneously. (2) Comparisons should take into account the statistical uncertainty induced by the choice of benchmark suite. (3) The robustness of the comparisons under small deviations in the underlying assumptions should be verifiable. To address (1), we propose to compare classifiers using a generalized stochastic dominance ordering (GSD) and present the GSD-front as an information-efficient alternative to the classical Pareto-front. For (2), we propose a consistent statistical estimator for the GSD-front and construct a statistical test for whether a (potentially new) classifier lies in the GSD-front of a set of state-of-the-art classifiers. For (3), we relax our proposed test using techniques from robust statistics and imprecise probabilities. We illustrate our concepts on the benchmark suite PMLB and on the platform OpenML. |
| title | Statistical Multicriteria Benchmarking via the GSD-Front |
| topic | Machine Learning Methodology 62G05, 62G35, 62G09, 62G10 G.3 |
| url | https://arxiv.org/abs/2406.03924 |