Quantile Learn-Then-Test: Quantile-Based Risk Control for Hyperparameter Optimization
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910540346949632 |
|---|---|
| author | Farzaneh, Amirmohammad Park, Sangwoo Simeone, Osvaldo |
| author_facet | Farzaneh, Amirmohammad Park, Sangwoo Simeone, Osvaldo |
| contents | The increasing adoption of Artificial Intelligence (AI) in engineering problems calls for the development of calibration methods capable of offering robust statistical reliability guarantees. The calibration of black box AI models is carried out via the optimization of hyperparameters dictating architecture, optimization, and/or inference configuration. Prior work has introduced learn-then-test (LTT), a calibration procedure for hyperparameter optimization (HPO) that provides statistical guarantees on average performance measures. Recognizing the importance of controlling risk-aware objectives in engineering contexts, this work introduces a variant of LTT that is designed to provide statistical guarantees on quantiles of a risk measure. We illustrate the practical advantages of this approach by applying the proposed algorithm to a radio access scheduling problem. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_17358 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Quantile Learn-Then-Test: Quantile-Based Risk Control for Hyperparameter Optimization Farzaneh, Amirmohammad Park, Sangwoo Simeone, Osvaldo Information Theory Machine Learning The increasing adoption of Artificial Intelligence (AI) in engineering problems calls for the development of calibration methods capable of offering robust statistical reliability guarantees. The calibration of black box AI models is carried out via the optimization of hyperparameters dictating architecture, optimization, and/or inference configuration. Prior work has introduced learn-then-test (LTT), a calibration procedure for hyperparameter optimization (HPO) that provides statistical guarantees on average performance measures. Recognizing the importance of controlling risk-aware objectives in engineering contexts, this work introduces a variant of LTT that is designed to provide statistical guarantees on quantiles of a risk measure. We illustrate the practical advantages of this approach by applying the proposed algorithm to a radio access scheduling problem. |
| title | Quantile Learn-Then-Test: Quantile-Based Risk Control for Hyperparameter Optimization |
| topic | Information Theory Machine Learning |
| url | https://arxiv.org/abs/2407.17358 |