Memorization in Attention-only Transformers
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866916647650983936 |
|---|---|
| author | Dana, Léo Pydi, Muni Sreenivas Chevaleyre, Yann |
| author_facet | Dana, Léo Pydi, Muni Sreenivas Chevaleyre, Yann |
| contents | Recent research has explored the memorization capacity of multi-head attention, but these findings are constrained by unrealistic limitations on the context size. We present a novel proof for language-based Transformers that extends the current hypothesis to any context size. Our approach improves upon the state-of-the-art by achieving more effective exact memorization with an attention layer, while also introducing the concept of approximate memorization of distributions. Through experimental validation, we demonstrate that our proposed bounds more accurately reflect the true memorization capacity of language models, and provide a precise comparison with prior work. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_10115 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Memorization in Attention-only Transformers Dana, Léo Pydi, Muni Sreenivas Chevaleyre, Yann Artificial Intelligence Computation and Language Recent research has explored the memorization capacity of multi-head attention, but these findings are constrained by unrealistic limitations on the context size. We present a novel proof for language-based Transformers that extends the current hypothesis to any context size. Our approach improves upon the state-of-the-art by achieving more effective exact memorization with an attention layer, while also introducing the concept of approximate memorization of distributions. Through experimental validation, we demonstrate that our proposed bounds more accurately reflect the true memorization capacity of language models, and provide a precise comparison with prior work. |
| title | Memorization in Attention-only Transformers |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2411.10115 |