Gated recurrent neural networks discover attention
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911772412215296 |
|---|---|
| author | Zucchet, Nicolas Kobayashi, Seijin Akram, Yassir von Oswald, Johannes Larcher, Maxime Steger, Angelika Sacramento, João |
| author_facet | Zucchet, Nicolas Kobayashi, Seijin Akram, Yassir von Oswald, Johannes Larcher, Maxime Steger, Angelika Sacramento, João |
| contents | Recent architectural developments have enabled recurrent neural networks (RNNs) to reach and even surpass the performance of Transformers on certain sequence modeling tasks. These modern RNNs feature a prominent design pattern: linear recurrent layers interconnected by feedforward paths with multiplicative gating. Here, we show how RNNs equipped with these two design elements can exactly implement (linear) self-attention, the main building block of Transformers. By reverse-engineering a set of trained RNNs, we find that gradient descent in practice discovers our construction. In particular, we examine RNNs trained to solve simple in-context learning tasks on which Transformers are known to excel and find that gradient descent instills in our RNNs the same attention-based in-context learning algorithm used by Transformers. Our findings highlight the importance of multiplicative interactions in neural networks and suggest that certain RNNs might be unexpectedly implementing attention under the hood. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2309_01775 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Gated recurrent neural networks discover attention Zucchet, Nicolas Kobayashi, Seijin Akram, Yassir von Oswald, Johannes Larcher, Maxime Steger, Angelika Sacramento, João Machine Learning Neural and Evolutionary Computing Recent architectural developments have enabled recurrent neural networks (RNNs) to reach and even surpass the performance of Transformers on certain sequence modeling tasks. These modern RNNs feature a prominent design pattern: linear recurrent layers interconnected by feedforward paths with multiplicative gating. Here, we show how RNNs equipped with these two design elements can exactly implement (linear) self-attention, the main building block of Transformers. By reverse-engineering a set of trained RNNs, we find that gradient descent in practice discovers our construction. In particular, we examine RNNs trained to solve simple in-context learning tasks on which Transformers are known to excel and find that gradient descent instills in our RNNs the same attention-based in-context learning algorithm used by Transformers. Our findings highlight the importance of multiplicative interactions in neural networks and suggest that certain RNNs might be unexpectedly implementing attention under the hood. |
| title | Gated recurrent neural networks discover attention |
| topic | Machine Learning Neural and Evolutionary Computing |
| url | https://arxiv.org/abs/2309.01775 |