DenseFormer: Enhancing Information Flow in Transformers via Depth Weighted Averaging
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911806589501440 |
|---|---|
| author | Pagliardini, Matteo Mohtashami, Amirkeivan Fleuret, Francois Jaggi, Martin |
| author_facet | Pagliardini, Matteo Mohtashami, Amirkeivan Fleuret, Francois Jaggi, Martin |
| contents | The transformer architecture by Vaswani et al. (2017) is now ubiquitous across application domains, from natural language processing to speech processing and image understanding. We propose DenseFormer, a simple modification to the standard architecture that improves the perplexity of the model without increasing its size -- adding a few thousand parameters for large-scale models in the 100B parameters range. Our approach relies on an additional averaging step after each transformer block, which computes a weighted average of current and past representations -- we refer to this operation as Depth-Weighted-Average (DWA). The learned DWA weights exhibit coherent patterns of information flow, revealing the strong and structured reuse of activations from distant layers. Experiments demonstrate that DenseFormer is more data efficient, reaching the same perplexity of much deeper transformer models, and that for the same perplexity, these new models outperform transformer baselines in terms of memory efficiency and inference time. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_02622 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | DenseFormer: Enhancing Information Flow in Transformers via Depth Weighted Averaging Pagliardini, Matteo Mohtashami, Amirkeivan Fleuret, Francois Jaggi, Martin Computation and Language Machine Learning The transformer architecture by Vaswani et al. (2017) is now ubiquitous across application domains, from natural language processing to speech processing and image understanding. We propose DenseFormer, a simple modification to the standard architecture that improves the perplexity of the model without increasing its size -- adding a few thousand parameters for large-scale models in the 100B parameters range. Our approach relies on an additional averaging step after each transformer block, which computes a weighted average of current and past representations -- we refer to this operation as Depth-Weighted-Average (DWA). The learned DWA weights exhibit coherent patterns of information flow, revealing the strong and structured reuse of activations from distant layers. Experiments demonstrate that DenseFormer is more data efficient, reaching the same perplexity of much deeper transformer models, and that for the same perplexity, these new models outperform transformer baselines in terms of memory efficiency and inference time. |
| title | DenseFormer: Enhancing Information Flow in Transformers via Depth Weighted Averaging |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2402.02622 |