Dense Associative Memory with Epanechnikov Energy
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866908803599958016 |
|---|---|
| author | Hoover, Benjamin Shi, Zhaoyang Balasubramanian, Krishnakumar Krotov, Dmitry Ram, Parikshit |
| author_facet | Hoover, Benjamin Shi, Zhaoyang Balasubramanian, Krishnakumar Krotov, Dmitry Ram, Parikshit |
| contents | We propose a novel energy function for Dense Associative Memory (DenseAM) networks, the log-sum-ReLU (LSR), inspired by optimal kernel density estimation. Unlike the common log-sum-exponential (LSE) function, LSR is based on the Epanechnikov kernel and enables exact memory retrieval with exponential capacity without requiring exponential separation functions. Moreover, it introduces abundant additional \emph{emergent} local minima while preserving perfect pattern recovery -- a characteristic previously unseen in DenseAM literature. Empirical results show that LSR energy has significantly more local minima (memories) that have comparable log-likelihood to LSE-based models. Analysis of LSR's emergent memories on image datasets reveals a degree of creativity and novelty, hinting at this method's potential for both large-scale memory storage and generative tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_10801 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Dense Associative Memory with Epanechnikov Energy Hoover, Benjamin Shi, Zhaoyang Balasubramanian, Krishnakumar Krotov, Dmitry Ram, Parikshit Machine Learning We propose a novel energy function for Dense Associative Memory (DenseAM) networks, the log-sum-ReLU (LSR), inspired by optimal kernel density estimation. Unlike the common log-sum-exponential (LSE) function, LSR is based on the Epanechnikov kernel and enables exact memory retrieval with exponential capacity without requiring exponential separation functions. Moreover, it introduces abundant additional \emph{emergent} local minima while preserving perfect pattern recovery -- a characteristic previously unseen in DenseAM literature. Empirical results show that LSR energy has significantly more local minima (memories) that have comparable log-likelihood to LSE-based models. Analysis of LSR's emergent memories on image datasets reveals a degree of creativity and novelty, hinting at this method's potential for both large-scale memory storage and generative tasks. |
| title | Dense Associative Memory with Epanechnikov Energy |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2506.10801 |