A Unified Information-Theoretic Framework for Meta-Learning Generalization
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911204788666368 |
|---|---|
| author | Wen, Wen Gong, Tieliang Dong, Yuxin Gao, Zeyu Liu, Yong-Jin |
| author_facet | Wen, Wen Gong, Tieliang Dong, Yuxin Gao, Zeyu Liu, Yong-Jin |
| contents | In recent years, information-theoretic generalization bounds have gained increasing attention for analyzing the generalization capabilities of meta-learning algorithms. However, existing results are confined to two-step bounds, failing to provide a sharper characterization of the meta-generalization gap that simultaneously accounts for environment-level and task-level dependencies. This paper addresses this fundamental limitation by developing a unified information-theoretic framework using a single-step derivation. The resulting meta-generalization bounds, expressed in terms of diverse information measures, exhibit substantial advantages over previous work, particularly in terms of tightness, scaling behavior associated with sampled tasks and samples per task, and computational tractability. Furthermore, through gradient covariance analysis, we provide new theoretical insights into the generalization properties of two classes of noisy and iterative meta-learning algorithms, where the meta-learner uses either the entire meta-training data (e.g., Reptile), or separate training and test data within the task (e.g., model agnostic meta-learning (MAML)). Numerical results validate the effectiveness of the derived bounds in capturing the generalization dynamics of meta-learning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_15559 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | A Unified Information-Theoretic Framework for Meta-Learning Generalization Wen, Wen Gong, Tieliang Dong, Yuxin Gao, Zeyu Liu, Yong-Jin Machine Learning Statistics Theory In recent years, information-theoretic generalization bounds have gained increasing attention for analyzing the generalization capabilities of meta-learning algorithms. However, existing results are confined to two-step bounds, failing to provide a sharper characterization of the meta-generalization gap that simultaneously accounts for environment-level and task-level dependencies. This paper addresses this fundamental limitation by developing a unified information-theoretic framework using a single-step derivation. The resulting meta-generalization bounds, expressed in terms of diverse information measures, exhibit substantial advantages over previous work, particularly in terms of tightness, scaling behavior associated with sampled tasks and samples per task, and computational tractability. Furthermore, through gradient covariance analysis, we provide new theoretical insights into the generalization properties of two classes of noisy and iterative meta-learning algorithms, where the meta-learner uses either the entire meta-training data (e.g., Reptile), or separate training and test data within the task (e.g., model agnostic meta-learning (MAML)). Numerical results validate the effectiveness of the derived bounds in capturing the generalization dynamics of meta-learning. |
| title | A Unified Information-Theoretic Framework for Meta-Learning Generalization |
| topic | Machine Learning Statistics Theory |
| url | https://arxiv.org/abs/2501.15559 |