Memory-Augmented Transformers: A Systematic Review from Neuroscience Principles to Enhanced Model Architectures
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912539569291264 |
|---|---|
| author | Omidi, Parsa Huang, Xingshuai Laborieux, Axel Nikpour, Bahareh Shi, Tianyu Eshaghi, Armaghan |
| author_facet | Omidi, Parsa Huang, Xingshuai Laborieux, Axel Nikpour, Bahareh Shi, Tianyu Eshaghi, Armaghan |
| contents | Memory is fundamental to intelligence, enabling learning, reasoning, and adaptability across biological and artificial systems. While Transformer architectures excel at sequence modeling, they face critical limitations in long-range context retention, continual learning, and knowledge integration. This review presents a unified framework bridging neuroscience principles, including dynamic multi-timescale memory, selective attention, and consolidation, with engineering advances in Memory-Augmented Transformers. We organize recent progress through three taxonomic dimensions: functional objectives (context extension, reasoning, knowledge integration, adaptation), memory representations (parameter-encoded, state-based, explicit, hybrid), and integration mechanisms (attention fusion, gated control, associative retrieval). Our analysis of core memory operations (reading, writing, forgetting, and capacity management) reveals a shift from static caches toward adaptive, test-time learning systems. We identify persistent challenges in scalability and interference, alongside emerging solutions including hierarchical buffering and surprise-gated updates. This synthesis provides a roadmap toward cognitively-inspired, lifelong-learning Transformer architectures. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_10824 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Memory-Augmented Transformers: A Systematic Review from Neuroscience Principles to Enhanced Model Architectures Omidi, Parsa Huang, Xingshuai Laborieux, Axel Nikpour, Bahareh Shi, Tianyu Eshaghi, Armaghan Machine Learning Computation and Language Memory is fundamental to intelligence, enabling learning, reasoning, and adaptability across biological and artificial systems. While Transformer architectures excel at sequence modeling, they face critical limitations in long-range context retention, continual learning, and knowledge integration. This review presents a unified framework bridging neuroscience principles, including dynamic multi-timescale memory, selective attention, and consolidation, with engineering advances in Memory-Augmented Transformers. We organize recent progress through three taxonomic dimensions: functional objectives (context extension, reasoning, knowledge integration, adaptation), memory representations (parameter-encoded, state-based, explicit, hybrid), and integration mechanisms (attention fusion, gated control, associative retrieval). Our analysis of core memory operations (reading, writing, forgetting, and capacity management) reveals a shift from static caches toward adaptive, test-time learning systems. We identify persistent challenges in scalability and interference, alongside emerging solutions including hierarchical buffering and surprise-gated updates. This synthesis provides a roadmap toward cognitively-inspired, lifelong-learning Transformer architectures. |
| title | Memory-Augmented Transformers: A Systematic Review from Neuroscience Principles to Enhanced Model Architectures |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2508.10824 |