Absolute abstraction: a renormalisation group approach
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915829183938560 |
|---|---|
| author | Caputo, Carlo Orientale Seiffert, Elias Frausin, Enrico Marsili, Matteo |
| author_facet | Caputo, Carlo Orientale Seiffert, Elias Frausin, Enrico Marsili, Matteo |
| contents | Abstraction is the process of extracting the essential features from raw data while ignoring irrelevant details. It is well known that abstraction emerges with depth in neural networks, where deep layers capture abstract characteristics of data by combining lower level features encoded in shallow layers (e.g. edges). Yet we argue that depth alone is not enough to develop truly abstract representations. We advocate that the level of abstraction crucially depends on how broad the training set is. We address the issue within a renormalisation group approach where a representation is expanded to encompass a broader set of data. We take the unique fixed point of this transformation -- the Hierarchical Feature Model -- as a candidate for a representation which is absolutely abstract. This theoretical picture is tested in numerical experiments based on Deep Belief Networks and auto-encoders trained on data of different breadth. These show that representations in neural networks approach the Hierarchical Feature Model as the data get broader and as depth increases, in agreement with theoretical predictions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_01656 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Absolute abstraction: a renormalisation group approach Caputo, Carlo Orientale Seiffert, Elias Frausin, Enrico Marsili, Matteo Machine Learning Disordered Systems and Neural Networks Data Analysis, Statistics and Probability Neurons and Cognition Abstraction is the process of extracting the essential features from raw data while ignoring irrelevant details. It is well known that abstraction emerges with depth in neural networks, where deep layers capture abstract characteristics of data by combining lower level features encoded in shallow layers (e.g. edges). Yet we argue that depth alone is not enough to develop truly abstract representations. We advocate that the level of abstraction crucially depends on how broad the training set is. We address the issue within a renormalisation group approach where a representation is expanded to encompass a broader set of data. We take the unique fixed point of this transformation -- the Hierarchical Feature Model -- as a candidate for a representation which is absolutely abstract. This theoretical picture is tested in numerical experiments based on Deep Belief Networks and auto-encoders trained on data of different breadth. These show that representations in neural networks approach the Hierarchical Feature Model as the data get broader and as depth increases, in agreement with theoretical predictions. |
| title | Absolute abstraction: a renormalisation group approach |
| topic | Machine Learning Disordered Systems and Neural Networks Data Analysis, Statistics and Probability Neurons and Cognition |
| url | https://arxiv.org/abs/2407.01656 |