Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2401.12819 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911763136512000 |
|---|---|
| author | Hay, Tamir David Wolf, Lior |
| author_facet | Hay, Tamir David Wolf, Lior |
| contents | In the pursuit of reducing the number of trainable parameters in deep transformer networks, we employ Reinforcement Learning to dynamically select layers during training and tie them together. Every few iterations, the RL agent is asked whether to train each layer $i$ independently or to copy the weights of a previous layer $j<i$. This facilitates weight sharing, reduces the number of trainable parameters, and also serves as an effective regularization technique. Experimental evaluations validate that our model modestly outperforms the baseline transformer model with regard to perplexity and drastically reduces the number of trainable parameters. In particular, the memory consumption during training is up to one order of magnitude less than the conventional training method. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_12819 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Dynamic Layer Tying for Parameter-Efficient Transformers Hay, Tamir David Wolf, Lior Machine Learning Artificial Intelligence In the pursuit of reducing the number of trainable parameters in deep transformer networks, we employ Reinforcement Learning to dynamically select layers during training and tie them together. Every few iterations, the RL agent is asked whether to train each layer $i$ independently or to copy the weights of a previous layer $j<i$. This facilitates weight sharing, reduces the number of trainable parameters, and also serves as an effective regularization technique. Experimental evaluations validate that our model modestly outperforms the baseline transformer model with regard to perplexity and drastically reduces the number of trainable parameters. In particular, the memory consumption during training is up to one order of magnitude less than the conventional training method. |
| title | Dynamic Layer Tying for Parameter-Efficient Transformers |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2401.12819 |