Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2405.15895 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909649456857088 |
|---|---|
| author | Malviya, Pranshu Huang, Jerry Baratin, Aristide Fournier, Quentin Chandar, Sarath |
| author_facet | Malviya, Pranshu Huang, Jerry Baratin, Aristide Fournier, Quentin Chandar, Sarath |
| contents | Determining the optimal model for a given task often requires training multiple models from scratch, which becomes impractical as dataset and model sizes grow. A more efficient alternative is to expand smaller pre-trained models, but this approach is underutilized due to a limited understanding of its impact on the training dynamics. Existing methods for quantifying this impact have notable limitations, including computation cost. To address this, we introduce a new perspective based on the loss landscape, which has been shown to contain a manifold of linearly connected minima. Specifically, we propose a metric that estimates the size of this manifold to study the impact of model expansion. Our experiments reveal a strong correlation between performance gains and our manifold metric, enabling more informed model comparison and offering a first step toward a geometry-driven approach for reliable model expansion. Notably, our metric outperforms other baselines, even when different types of expansion with equivalent number of parameters are applied to a model. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_15895 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Manifold Metric: A Loss Landscape Approach for Predicting Model Performance Malviya, Pranshu Huang, Jerry Baratin, Aristide Fournier, Quentin Chandar, Sarath Machine Learning Determining the optimal model for a given task often requires training multiple models from scratch, which becomes impractical as dataset and model sizes grow. A more efficient alternative is to expand smaller pre-trained models, but this approach is underutilized due to a limited understanding of its impact on the training dynamics. Existing methods for quantifying this impact have notable limitations, including computation cost. To address this, we introduce a new perspective based on the loss landscape, which has been shown to contain a manifold of linearly connected minima. Specifically, we propose a metric that estimates the size of this manifold to study the impact of model expansion. Our experiments reveal a strong correlation between performance gains and our manifold metric, enabling more informed model comparison and offering a first step toward a geometry-driven approach for reliable model expansion. Notably, our metric outperforms other baselines, even when different types of expansion with equivalent number of parameters are applied to a model. |
| title | Manifold Metric: A Loss Landscape Approach for Predicting Model Performance |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2405.15895 |