Enhancing Length Extrapolation in Sequential Models with Pointer-Augmented Neural Memory
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866929318561579008 |
|---|---|
| author | Le, Hung Nguyen, Dung Do, Kien Venkatesh, Svetha Tran, Truyen |
| author_facet | Le, Hung Nguyen, Dung Do, Kien Venkatesh, Svetha Tran, Truyen |
| contents | We propose Pointer-Augmented Neural Memory (PANM) to help neural networks understand and apply symbol processing to new, longer sequences of data. PANM integrates an external neural memory that uses novel physical addresses and pointer manipulation techniques to mimic human and computer symbol processing abilities. PANM facilitates pointer assignment, dereference, and arithmetic by explicitly using physical pointers to access memory content. Remarkably, it can learn to perform these operations through end-to-end training on sequence data, powering various sequential models. Our experiments demonstrate PANM's exceptional length extrapolating capabilities and improved performance in tasks that require symbol processing, such as algorithmic reasoning and Dyck language recognition. PANM helps Transformer achieve up to 100% generalization accuracy in compositional learning tasks and significantly better results in mathematical reasoning, question answering and machine translation tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_11870 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Enhancing Length Extrapolation in Sequential Models with Pointer-Augmented Neural Memory Le, Hung Nguyen, Dung Do, Kien Venkatesh, Svetha Tran, Truyen Machine Learning Computation and Language We propose Pointer-Augmented Neural Memory (PANM) to help neural networks understand and apply symbol processing to new, longer sequences of data. PANM integrates an external neural memory that uses novel physical addresses and pointer manipulation techniques to mimic human and computer symbol processing abilities. PANM facilitates pointer assignment, dereference, and arithmetic by explicitly using physical pointers to access memory content. Remarkably, it can learn to perform these operations through end-to-end training on sequence data, powering various sequential models. Our experiments demonstrate PANM's exceptional length extrapolating capabilities and improved performance in tasks that require symbol processing, such as algorithmic reasoning and Dyck language recognition. PANM helps Transformer achieve up to 100% generalization accuracy in compositional learning tasks and significantly better results in mathematical reasoning, question answering and machine translation tasks. |
| title | Enhancing Length Extrapolation in Sequential Models with Pointer-Augmented Neural Memory |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2404.11870 |