Non-Vacuous Generalization Bounds: Can Rescaling Invariances Help?
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911185852432384 |
|---|---|
| author | Rouchouse, Damien Gonon, Antoine Gribonval, Rémi Guedj, Benjamin |
| author_facet | Rouchouse, Damien Gonon, Antoine Gribonval, Rémi Guedj, Benjamin |
| contents | A central challenge in understanding generalization is to obtain non-vacuous guarantees that go beyond worst-case complexity over data or weight space. Among existing approaches, PAC-Bayes bounds stand out as they can provide tight, data-dependent guarantees even for large networks. However, in ReLU networks, rescaling invariances mean that different weight distributions can represent the same function while leading to arbitrarily different PAC-Bayes complexities. We propose to study PAC-Bayes bounds in an invariant, lifted representation that resolves this discrepancy. This paper explores both the guarantees provided by this approach (invariance, tighter bounds via data processing) and the algorithmic aspects of KL-based rescaling-invariant PAC-Bayes bounds. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_26149 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Non-Vacuous Generalization Bounds: Can Rescaling Invariances Help? Rouchouse, Damien Gonon, Antoine Gribonval, Rémi Guedj, Benjamin Machine Learning A central challenge in understanding generalization is to obtain non-vacuous guarantees that go beyond worst-case complexity over data or weight space. Among existing approaches, PAC-Bayes bounds stand out as they can provide tight, data-dependent guarantees even for large networks. However, in ReLU networks, rescaling invariances mean that different weight distributions can represent the same function while leading to arbitrarily different PAC-Bayes complexities. We propose to study PAC-Bayes bounds in an invariant, lifted representation that resolves this discrepancy. This paper explores both the guarantees provided by this approach (invariance, tighter bounds via data processing) and the algorithmic aspects of KL-based rescaling-invariant PAC-Bayes bounds. |
| title | Non-Vacuous Generalization Bounds: Can Rescaling Invariances Help? |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2509.26149 |