Adaptive Learn-then-Test: Statistically Valid and Efficient Hyperparameter Selection
Fuente:
arXiv
Guardado en:
| Autores principales: | , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917907067305984 |
|---|---|
| author | Zecchin, Matteo Park, Sangwoo Simeone, Osvaldo |
| author_facet | Zecchin, Matteo Park, Sangwoo Simeone, Osvaldo |
| contents | We introduce adaptive learn-then-test (aLTT), an efficient hyperparameter selection procedure that provides finite-sample statistical guarantees on the population risk of AI models. Unlike the existing learn-then-test (LTT) technique, which relies on conventional p-value-based multiple hypothesis testing (MHT), aLTT implements sequential data-dependent MHT with early termination by leveraging e-processes. As a result, aLTT can reduce the number of testing rounds, making it particularly well-suited for scenarios in which testing is costly or presents safety risks. Apart from maintaining statistical validity, in applications such as online policy selection for offline reinforcement learning and prompt engineering, aLTT is shown to achieve the same performance as LTT while requiring only a fraction of the testing rounds. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_15844 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Adaptive Learn-then-Test: Statistically Valid and Efficient Hyperparameter Selection Zecchin, Matteo Park, Sangwoo Simeone, Osvaldo Machine Learning Artificial Intelligence Information Theory Methodology We introduce adaptive learn-then-test (aLTT), an efficient hyperparameter selection procedure that provides finite-sample statistical guarantees on the population risk of AI models. Unlike the existing learn-then-test (LTT) technique, which relies on conventional p-value-based multiple hypothesis testing (MHT), aLTT implements sequential data-dependent MHT with early termination by leveraging e-processes. As a result, aLTT can reduce the number of testing rounds, making it particularly well-suited for scenarios in which testing is costly or presents safety risks. Apart from maintaining statistical validity, in applications such as online policy selection for offline reinforcement learning and prompt engineering, aLTT is shown to achieve the same performance as LTT while requiring only a fraction of the testing rounds. |
| title | Adaptive Learn-then-Test: Statistically Valid and Efficient Hyperparameter Selection |
| topic | Machine Learning Artificial Intelligence Information Theory Methodology |
| url | https://arxiv.org/abs/2409.15844 |