Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2404.09173 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929337084674048 |
|---|---|
| author | Hwang, Dongseong Wang, Weiran Huo, Zhuoyuan Sim, Khe Chai Mengibar, Pedro Moreno |
| author_facet | Hwang, Dongseong Wang, Weiran Huo, Zhuoyuan Sim, Khe Chai Mengibar, Pedro Moreno |
| contents | While Transformers have revolutionized deep learning, their quadratic attention complexity hinders their ability to process infinitely long inputs. We propose Feedback Attention Memory (FAM), a novel Transformer architecture that leverages a feedback loop to enable the network to attend to its own latent representations. This design fosters the emergence of working memory within the Transformer, allowing it to process indefinitely long sequences. TransformerFAM requires no additional weights, enabling seamless integration with pre-trained models. Our experiments show that TransformerFAM significantly improves Transformer performance on long-context tasks across various model sizes (1B, 8B, and 24B). These results showcase the potential to empower Large Language Models (LLMs) to process sequences of unlimited length. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_09173 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | TransformerFAM: Feedback attention is working memory Hwang, Dongseong Wang, Weiran Huo, Zhuoyuan Sim, Khe Chai Mengibar, Pedro Moreno Machine Learning Artificial Intelligence Computation and Language While Transformers have revolutionized deep learning, their quadratic attention complexity hinders their ability to process infinitely long inputs. We propose Feedback Attention Memory (FAM), a novel Transformer architecture that leverages a feedback loop to enable the network to attend to its own latent representations. This design fosters the emergence of working memory within the Transformer, allowing it to process indefinitely long sequences. TransformerFAM requires no additional weights, enabling seamless integration with pre-trained models. Our experiments show that TransformerFAM significantly improves Transformer performance on long-context tasks across various model sizes (1B, 8B, and 24B). These results showcase the potential to empower Large Language Models (LLMs) to process sequences of unlimited length. |
| title | TransformerFAM: Feedback attention is working memory |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2404.09173 |