Memory Layers at Scale
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909436325396480 |
|---|---|
| author | Berges, Vincent-Pierre Oğuz, Barlas Haziza, Daniel Yih, Wen-tau Zettlemoyer, Luke Ghosh, Gargi |
| author_facet | Berges, Vincent-Pierre Oğuz, Barlas Haziza, Daniel Yih, Wen-tau Zettlemoyer, Luke Ghosh, Gargi |
| contents | Memory layers use a trainable key-value lookup mechanism to add extra parameters to a model without increasing FLOPs. Conceptually, sparsely activated memory layers complement compute-heavy dense feed-forward layers, providing dedicated capacity to store and retrieve information cheaply. This work takes memory layers beyond proof-of-concept, proving their utility at contemporary scale. On downstream tasks, language models augmented with our improved memory layer outperform dense models with more than twice the computation budget, as well as mixture-of-expert models when matched for both compute and parameters. We find gains are especially pronounced for factual tasks. We provide a fully parallelizable memory layer implementation, demonstrating scaling laws with up to 128B memory parameters, pretrained to 1 trillion tokens, comparing to base models with up to 8B parameters. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_09764 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Memory Layers at Scale Berges, Vincent-Pierre Oğuz, Barlas Haziza, Daniel Yih, Wen-tau Zettlemoyer, Luke Ghosh, Gargi Computation and Language Artificial Intelligence Memory layers use a trainable key-value lookup mechanism to add extra parameters to a model without increasing FLOPs. Conceptually, sparsely activated memory layers complement compute-heavy dense feed-forward layers, providing dedicated capacity to store and retrieve information cheaply. This work takes memory layers beyond proof-of-concept, proving their utility at contemporary scale. On downstream tasks, language models augmented with our improved memory layer outperform dense models with more than twice the computation budget, as well as mixture-of-expert models when matched for both compute and parameters. We find gains are especially pronounced for factual tasks. We provide a fully parallelizable memory layer implementation, demonstrating scaling laws with up to 128B memory parameters, pretrained to 1 trillion tokens, comparing to base models with up to 8B parameters. |
| title | Memory Layers at Scale |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2412.09764 |