Bayes meets Bernstein at the Meta Level: an Analysis of Fast Rates in Meta-Learning with PAC-Bayes
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2023
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912250993836032 |
|---|---|
| author | Riou, Charles Alquier, Pierre Chérief-Abdellatif, Badr-Eddine |
| author_facet | Riou, Charles Alquier, Pierre Chérief-Abdellatif, Badr-Eddine |
| contents | Bernstein's condition is a key assumption that guarantees fast rates in machine learning. For example, the Gibbs algorithm with prior $π$ has an excess risk in $O(d_π/n)$, as opposed to the standard $O(\sqrt{d_π/n})$, where $n$ denotes the number of observations and $d_π$ is a complexity parameter which depends on the prior $π$. In this paper, we examine the Gibbs algorithm in the context of meta-learning, i.e., when learning the prior $π$ from $T$ tasks (with $n$ observations each) generated by a meta distribution. Our main result is that Bernstein's condition always holds at the meta level, regardless of its validity at the observation level. This implies that the additional cost to learn the Gibbs prior $π$, which will reduce the term $d_π$ across tasks, is in $O(1/T)$, instead of the expected $O(1/\sqrt{T})$. We further illustrate how this result improves on standard rates in three different settings: discrete priors, Gaussian priors and mixture of Gaussians priors. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2302_11709 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Bayes meets Bernstein at the Meta Level: an Analysis of Fast Rates in Meta-Learning with PAC-Bayes Riou, Charles Alquier, Pierre Chérief-Abdellatif, Badr-Eddine Machine Learning Bernstein's condition is a key assumption that guarantees fast rates in machine learning. For example, the Gibbs algorithm with prior $π$ has an excess risk in $O(d_π/n)$, as opposed to the standard $O(\sqrt{d_π/n})$, where $n$ denotes the number of observations and $d_π$ is a complexity parameter which depends on the prior $π$. In this paper, we examine the Gibbs algorithm in the context of meta-learning, i.e., when learning the prior $π$ from $T$ tasks (with $n$ observations each) generated by a meta distribution. Our main result is that Bernstein's condition always holds at the meta level, regardless of its validity at the observation level. This implies that the additional cost to learn the Gibbs prior $π$, which will reduce the term $d_π$ across tasks, is in $O(1/T)$, instead of the expected $O(1/\sqrt{T})$. We further illustrate how this result improves on standard rates in three different settings: discrete priors, Gaussian priors and mixture of Gaussians priors. |
| title | Bayes meets Bernstein at the Meta Level: an Analysis of Fast Rates in Meta-Learning with PAC-Bayes |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2302.11709 |