TransFusion: Covariate-Shift Robust Transfer Learning for High-Dimensional Regression
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911821572603904 |
|---|---|
| author | He, Zelin Sun, Ying Liu, Jingyuan Li, Runze |
| author_facet | He, Zelin Sun, Ying Liu, Jingyuan Li, Runze |
| contents | The main challenge that sets transfer learning apart from traditional supervised learning is the distribution shift, reflected as the shift between the source and target models and that between the marginal covariate distributions. In this work, we tackle model shifts in the presence of covariate shifts in the high-dimensional regression setting. Specifically, we propose a two-step method with a novel fused-regularizer that effectively leverages samples from source tasks to improve the learning performance on a target task with limited samples. Nonasymptotic bound is provided for the estimation error of the target model, showing the robustness of the proposed method to covariate shifts. We further establish conditions under which the estimator is minimax-optimal. Additionally, we extend the method to a distributed setting, allowing for a pretraining-finetuning strategy, requiring just one round of communication while retaining the estimation rate of the centralized version. Numerical tests validate our theory, highlighting the method's robustness to covariate shifts. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_01153 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | TransFusion: Covariate-Shift Robust Transfer Learning for High-Dimensional Regression He, Zelin Sun, Ying Liu, Jingyuan Li, Runze Machine Learning Distributed, Parallel, and Cluster Computing Statistics Theory Methodology The main challenge that sets transfer learning apart from traditional supervised learning is the distribution shift, reflected as the shift between the source and target models and that between the marginal covariate distributions. In this work, we tackle model shifts in the presence of covariate shifts in the high-dimensional regression setting. Specifically, we propose a two-step method with a novel fused-regularizer that effectively leverages samples from source tasks to improve the learning performance on a target task with limited samples. Nonasymptotic bound is provided for the estimation error of the target model, showing the robustness of the proposed method to covariate shifts. We further establish conditions under which the estimator is minimax-optimal. Additionally, we extend the method to a distributed setting, allowing for a pretraining-finetuning strategy, requiring just one round of communication while retaining the estimation rate of the centralized version. Numerical tests validate our theory, highlighting the method's robustness to covariate shifts. |
| title | TransFusion: Covariate-Shift Robust Transfer Learning for High-Dimensional Regression |
| topic | Machine Learning Distributed, Parallel, and Cluster Computing Statistics Theory Methodology |
| url | https://arxiv.org/abs/2404.01153 |