Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2305.15786 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912765842554880 |
|---|---|
| author | Hasson, Hilaf Maddix, Danielle C. Wang, Yuyang Gupta, Gaurav Park, Youngsuk |
| author_facet | Hasson, Hilaf Maddix, Danielle C. Wang, Yuyang Gupta, Gaurav Park, Youngsuk |
| contents | Ensembling is among the most popular tools in machine learning (ML) due to its effectiveness in minimizing variance and thus improving generalization. Most ensembling methods for black-box base learners fall under the umbrella of "stacked generalization," namely training an ML algorithm that takes the inferences from the base learners as input. While stacking has been widely applied in practice, its theoretical properties are poorly understood. In this paper, we prove a novel result, showing that choosing the best stacked generalization from a (finite or finite-dimensional) family of stacked generalizations based on cross-validated performance does not perform "much worse" than the oracle best. Our result strengthens and significantly extends the results in Van der Laan et al. (2007). Inspired by the theoretical analysis, we further propose a particular family of stacked generalizations in the context of probabilistic forecasting, each one with a different sensitivity for how much the ensemble weights are allowed to vary across items, timestamps in the forecast horizon, and quantiles. Experimental results demonstrate the performance gain of the proposed method. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2305_15786 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Theoretical Guarantees of Learning Ensembling Strategies with Applications to Time Series Forecasting Hasson, Hilaf Maddix, Danielle C. Wang, Yuyang Gupta, Gaurav Park, Youngsuk Machine Learning Statistics Theory Ensembling is among the most popular tools in machine learning (ML) due to its effectiveness in minimizing variance and thus improving generalization. Most ensembling methods for black-box base learners fall under the umbrella of "stacked generalization," namely training an ML algorithm that takes the inferences from the base learners as input. While stacking has been widely applied in practice, its theoretical properties are poorly understood. In this paper, we prove a novel result, showing that choosing the best stacked generalization from a (finite or finite-dimensional) family of stacked generalizations based on cross-validated performance does not perform "much worse" than the oracle best. Our result strengthens and significantly extends the results in Van der Laan et al. (2007). Inspired by the theoretical analysis, we further propose a particular family of stacked generalizations in the context of probabilistic forecasting, each one with a different sensitivity for how much the ensemble weights are allowed to vary across items, timestamps in the forecast horizon, and quantiles. Experimental results demonstrate the performance gain of the proposed method. |
| title | Theoretical Guarantees of Learning Ensembling Strategies with Applications to Time Series Forecasting |
| topic | Machine Learning Statistics Theory |
| url | https://arxiv.org/abs/2305.15786 |