Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2410.03156 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909335824629760 |
|---|---|
| author | Chen, Yinpeng Hutchins, DeLesley Jansen, Aren Zhmoginov, Andrey Racz, David Andersen, Jesper |
| author_facet | Chen, Yinpeng Hutchins, DeLesley Jansen, Aren Zhmoginov, Andrey Racz, David Andersen, Jesper |
| contents | We present MELODI, a novel memory architecture designed to efficiently process long documents using short context windows. The key principle behind MELODI is to represent short-term and long-term memory as a hierarchical compression scheme across both network layers and context windows. Specifically, the short-term memory is achieved through recurrent compression of context windows across multiple layers, ensuring smooth transitions between windows. In contrast, the long-term memory performs further compression within a single middle layer and aggregates information across context windows, effectively consolidating crucial information from the entire history. Compared to a strong baseline - the Memorizing Transformer employing dense attention over a large long-term memory (64K key-value pairs) - our method demonstrates superior performance on various long-context datasets while remarkably reducing the memory footprint by a factor of 8. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_03156 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MELODI: Exploring Memory Compression for Long Contexts Chen, Yinpeng Hutchins, DeLesley Jansen, Aren Zhmoginov, Andrey Racz, David Andersen, Jesper Machine Learning Artificial Intelligence We present MELODI, a novel memory architecture designed to efficiently process long documents using short context windows. The key principle behind MELODI is to represent short-term and long-term memory as a hierarchical compression scheme across both network layers and context windows. Specifically, the short-term memory is achieved through recurrent compression of context windows across multiple layers, ensuring smooth transitions between windows. In contrast, the long-term memory performs further compression within a single middle layer and aggregates information across context windows, effectively consolidating crucial information from the entire history. Compared to a strong baseline - the Memorizing Transformer employing dense attention over a large long-term memory (64K key-value pairs) - our method demonstrates superior performance on various long-context datasets while remarkably reducing the memory footprint by a factor of 8. |
| title | MELODI: Exploring Memory Compression for Long Contexts |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2410.03156 |