The Exponential Capacity of Dense Associative Memories
Fuente:
arXiv
Salvato in:
| Autori principali: | , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2023
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917588345290752 |
|---|---|
| author | Lucibello, Carlo Mézard, Marc |
| author_facet | Lucibello, Carlo Mézard, Marc |
| contents | Recent generalizations of the Hopfield model of associative memories are able to store a number $P$ of random patterns that grows exponentially with the number $N$ of neurons, $P=\exp(αN)$. Besides the huge storage capacity, another interesting feature of these networks is their connection to the attention mechanism which is part of the Transformer architectures widely applied in deep learning. In this work, we study a generic family of pattern ensembles using a statistical mechanics analysis which gives exact asymptotic thresholds for the retrieval of a typical pattern, $α_1$, and lower bounds for the maximum of the load $α$ for which all patterns can be retrieved, $α_c$, as well as sizes of attraction basins. We discuss in detail the cases of Gaussian and spherical patterns, and show that they display rich and qualitatively different phase diagrams. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2304_14964 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | The Exponential Capacity of Dense Associative Memories Lucibello, Carlo Mézard, Marc Disordered Systems and Neural Networks Information Theory Recent generalizations of the Hopfield model of associative memories are able to store a number $P$ of random patterns that grows exponentially with the number $N$ of neurons, $P=\exp(αN)$. Besides the huge storage capacity, another interesting feature of these networks is their connection to the attention mechanism which is part of the Transformer architectures widely applied in deep learning. In this work, we study a generic family of pattern ensembles using a statistical mechanics analysis which gives exact asymptotic thresholds for the retrieval of a typical pattern, $α_1$, and lower bounds for the maximum of the load $α$ for which all patterns can be retrieved, $α_c$, as well as sizes of attraction basins. We discuss in detail the cases of Gaussian and spherical patterns, and show that they display rich and qualitatively different phase diagrams. |
| title | The Exponential Capacity of Dense Associative Memories |
| topic | Disordered Systems and Neural Networks Information Theory |
| url | https://arxiv.org/abs/2304.14964 |