M+: Extending MemoryLLM with Scalable Long-Term Memory
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912404043988992 |
|---|---|
| author | Wang, Yu Krotov, Dmitry Hu, Yuanzhe Gao, Yifan Zhou, Wangchunshu McAuley, Julian Gutfreund, Dan Feris, Rogerio He, Zexue |
| author_facet | Wang, Yu Krotov, Dmitry Hu, Yuanzhe Gao, Yifan Zhou, Wangchunshu McAuley, Julian Gutfreund, Dan Feris, Rogerio He, Zexue |
| contents | Equipping large language models (LLMs) with latent-space memory has attracted increasing attention as they can extend the context window of existing language models. However, retaining information from the distant past remains a challenge. For example, MemoryLLM (Wang et al., 2024a), as a representative work with latent-space memory, compresses past information into hidden states across all layers, forming a memory pool of 1B parameters. While effective for sequence lengths up to 16k tokens, it struggles to retain knowledge beyond 20k tokens. In this work, we address this limitation by introducing M+, a memory-augmented model based on MemoryLLM that significantly enhances long-term information retention. M+ integrates a long-term memory mechanism with a co-trained retriever, dynamically retrieving relevant information during text generation. We evaluate M+ on diverse benchmarks, including long-context understanding and knowledge retention tasks. Experimental results show that M+ significantly outperforms MemoryLLM and recent strong baselines, extending knowledge retention from under 20k to over 160k tokens with similar GPU memory overhead. We open-source our code at https://github.com/wangyu-ustc/MemoryLLM |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_00592 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | M+: Extending MemoryLLM with Scalable Long-Term Memory Wang, Yu Krotov, Dmitry Hu, Yuanzhe Gao, Yifan Zhou, Wangchunshu McAuley, Julian Gutfreund, Dan Feris, Rogerio He, Zexue Computation and Language Equipping large language models (LLMs) with latent-space memory has attracted increasing attention as they can extend the context window of existing language models. However, retaining information from the distant past remains a challenge. For example, MemoryLLM (Wang et al., 2024a), as a representative work with latent-space memory, compresses past information into hidden states across all layers, forming a memory pool of 1B parameters. While effective for sequence lengths up to 16k tokens, it struggles to retain knowledge beyond 20k tokens. In this work, we address this limitation by introducing M+, a memory-augmented model based on MemoryLLM that significantly enhances long-term information retention. M+ integrates a long-term memory mechanism with a co-trained retriever, dynamically retrieving relevant information during text generation. We evaluate M+ on diverse benchmarks, including long-context understanding and knowledge retention tasks. Experimental results show that M+ significantly outperforms MemoryLLM and recent strong baselines, extending knowledge retention from under 20k to over 160k tokens with similar GPU memory overhead. We open-source our code at https://github.com/wangyu-ustc/MemoryLLM |
| title | M+: Extending MemoryLLM with Scalable Long-Term Memory |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2502.00592 |