Scaling Laws for Associative Memories
Fuente:
arXiv
Guardado en:
| Autores principales: | , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2023
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917594725875712 |
|---|---|
| author | Cabannes, Vivien Dohmatob, Elvis Bietti, Alberto |
| author_facet | Cabannes, Vivien Dohmatob, Elvis Bietti, Alberto |
| contents | Learning arguably involves the discovery and memorization of abstract rules. The aim of this paper is to study associative memory mechanisms. Our model is based on high-dimensional matrices consisting of outer products of embeddings, which relates to the inner layers of transformer language models. We derive precise scaling laws with respect to sample size and parameter size, and discuss the statistical efficiency of different estimators, including optimization-based algorithms. We provide extensive numerical experiments to validate and interpret theoretical results, including fine-grained visualizations of the stored memory associations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_02984 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Scaling Laws for Associative Memories Cabannes, Vivien Dohmatob, Elvis Bietti, Alberto Machine Learning Artificial Intelligence Computation and Language Neural and Evolutionary Computing I.2.6; G.1.6 Learning arguably involves the discovery and memorization of abstract rules. The aim of this paper is to study associative memory mechanisms. Our model is based on high-dimensional matrices consisting of outer products of embeddings, which relates to the inner layers of transformer language models. We derive precise scaling laws with respect to sample size and parameter size, and discuss the statistical efficiency of different estimators, including optimization-based algorithms. We provide extensive numerical experiments to validate and interpret theoretical results, including fine-grained visualizations of the stored memory associations. |
| title | Scaling Laws for Associative Memories |
| topic | Machine Learning Artificial Intelligence Computation and Language Neural and Evolutionary Computing I.2.6; G.1.6 |
| url | https://arxiv.org/abs/2310.02984 |