Mapping of attention mechanisms to a generalized Potts model
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2023
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916207295201280 |
|---|---|
| author | Rende, Riccardo Gerace, Federica Laio, Alessandro Goldt, Sebastian |
| author_facet | Rende, Riccardo Gerace, Federica Laio, Alessandro Goldt, Sebastian |
| contents | Transformers are neural networks that revolutionized natural language processing and machine learning. They process sequences of inputs, like words, using a mechanism called self-attention, which is trained via masked language modeling (MLM). In MLM, a word is randomly masked in an input sequence, and the network is trained to predict the missing word. Despite the practical success of transformers, it remains unclear what type of data distribution self-attention can learn efficiently. Here, we show analytically that if one decouples the treatment of word positions and embeddings, a single layer of self-attention learns the conditionals of a generalized Potts model with interactions between sites and Potts colors. Moreover, we show that training this neural network is exactly equivalent to solving the inverse Potts problem by the so-called pseudo-likelihood method, well known in statistical physics. Using this mapping, we compute the generalization error of self-attention in a model scenario analytically using the replica method. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2304_07235 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Mapping of attention mechanisms to a generalized Potts model Rende, Riccardo Gerace, Federica Laio, Alessandro Goldt, Sebastian Disordered Systems and Neural Networks Statistical Mechanics Computation and Language Machine Learning Transformers are neural networks that revolutionized natural language processing and machine learning. They process sequences of inputs, like words, using a mechanism called self-attention, which is trained via masked language modeling (MLM). In MLM, a word is randomly masked in an input sequence, and the network is trained to predict the missing word. Despite the practical success of transformers, it remains unclear what type of data distribution self-attention can learn efficiently. Here, we show analytically that if one decouples the treatment of word positions and embeddings, a single layer of self-attention learns the conditionals of a generalized Potts model with interactions between sites and Potts colors. Moreover, we show that training this neural network is exactly equivalent to solving the inverse Potts problem by the so-called pseudo-likelihood method, well known in statistical physics. Using this mapping, we compute the generalization error of self-attention in a model scenario analytically using the replica method. |
| title | Mapping of attention mechanisms to a generalized Potts model |
| topic | Disordered Systems and Neural Networks Statistical Mechanics Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2304.07235 |