Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2410.13577 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916779840765952 |
|---|---|
| author | Leblanc, Benjamin Bazinet, Mathieu D'Amours, Nathaniel Drouin, Alexandre Germain, Pascal |
| author_facet | Leblanc, Benjamin Bazinet, Mathieu D'Amours, Nathaniel Drouin, Alexandre Germain, Pascal |
| contents | Both PAC-Bayesian and Sample Compress learning frameworks are instrumental for deriving tight (non-vacuous) generalization bounds for neural networks. We leverage these results in a meta-learning scheme, relying on a hypernetwork that outputs the parameters of a downstream predictor from a dataset input. The originality of our approach lies in the investigated hypernetwork architectures that encode the dataset before decoding the parameters: (1) a PAC-Bayesian encoder that expresses a posterior distribution over a latent space, (2) a Sample Compress encoder that selects a small sample of the dataset input along with a message from a discrete set, and (3) a hybrid between both approaches motivated by a new Sample Compress theorem handling continuous messages. The latter theorem exploits the pivotal information transiting at the encoder-decoder junction in order to compute generalization guarantees for each downstream predictor obtained by our meta-learning scheme. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_13577 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Generalization Bounds via Meta-Learned Model Representations: PAC-Bayes and Sample Compression Hypernetworks Leblanc, Benjamin Bazinet, Mathieu D'Amours, Nathaniel Drouin, Alexandre Germain, Pascal Machine Learning Both PAC-Bayesian and Sample Compress learning frameworks are instrumental for deriving tight (non-vacuous) generalization bounds for neural networks. We leverage these results in a meta-learning scheme, relying on a hypernetwork that outputs the parameters of a downstream predictor from a dataset input. The originality of our approach lies in the investigated hypernetwork architectures that encode the dataset before decoding the parameters: (1) a PAC-Bayesian encoder that expresses a posterior distribution over a latent space, (2) a Sample Compress encoder that selects a small sample of the dataset input along with a message from a discrete set, and (3) a hybrid between both approaches motivated by a new Sample Compress theorem handling continuous messages. The latter theorem exploits the pivotal information transiting at the encoder-decoder junction in order to compute generalization guarantees for each downstream predictor obtained by our meta-learning scheme. |
| title | Generalization Bounds via Meta-Learned Model Representations: PAC-Bayes and Sample Compression Hypernetworks |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2410.13577 |