Characterising the Inductive Biases of Neural Networks on Boolean Data
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866908384827015168 |
|---|---|
| author | Mingard, Chris Seier, Lukas Göring, Niclas Badelita, Andrei-Vlad London, Charles Louis, Ard |
| author_facet | Mingard, Chris Seier, Lukas Göring, Niclas Badelita, Andrei-Vlad London, Charles Louis, Ard |
| contents | Deep neural networks are renowned for their ability to generalise well across diverse tasks, even when heavily overparameterized. Existing works offer only partial explanations (for example, the NTK-based task-model alignment explanation neglects feature learning). Here, we provide an end-to-end, analytically tractable case study that links a network's inductive prior, its training dynamics including feature learning, and its eventual generalisation. Specifically, we exploit the one-to-one correspondence between depth-2 discrete fully connected networks and disjunctive normal form (DNF) formulas by training on Boolean functions. Under a Monte Carlo learning algorithm, our model exhibits predictable training dynamics and the emergence of interpretable features. This framework allows us to trace, in detail, how inductive bias and feature formation drive generalisation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_24060 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Characterising the Inductive Biases of Neural Networks on Boolean Data Mingard, Chris Seier, Lukas Göring, Niclas Badelita, Andrei-Vlad London, Charles Louis, Ard Machine Learning Deep neural networks are renowned for their ability to generalise well across diverse tasks, even when heavily overparameterized. Existing works offer only partial explanations (for example, the NTK-based task-model alignment explanation neglects feature learning). Here, we provide an end-to-end, analytically tractable case study that links a network's inductive prior, its training dynamics including feature learning, and its eventual generalisation. Specifically, we exploit the one-to-one correspondence between depth-2 discrete fully connected networks and disjunctive normal form (DNF) formulas by training on Boolean functions. Under a Monte Carlo learning algorithm, our model exhibits predictable training dynamics and the emergence of interpretable features. This framework allows us to trace, in detail, how inductive bias and feature formation drive generalisation. |
| title | Characterising the Inductive Biases of Neural Networks on Boolean Data |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2505.24060 |