Salvato in:
| Autori principali: | , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2411.13512 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916490087759872 |
|---|---|
| author | Aarts, Gert Hajizadeh, Ouraman Lucini, Biagio Park, Chanju |
| author_facet | Aarts, Gert Hajizadeh, Ouraman Lucini, Biagio Park, Chanju |
| contents | During training, weight matrices in machine learning architectures are updated using stochastic gradient descent or variations thereof. In this contribution we employ concepts of random matrix theory to analyse the resulting stochastic matrix dynamics. We first demonstrate that the dynamics can generically be described using Dyson Brownian motion, leading to e.g. eigenvalue repulsion. The level of stochasticity is shown to depend on the ratio of the learning rate and the mini-batch size, explaining the empirically observed linear scaling rule. We verify this linear scaling in the restricted Boltzmann machine. Subsequently we study weight matrix dynamics in transformers (a nano-GPT), following the evolution from a Marchenko-Pastur distribution for eigenvalues at initialisation to a combination with additional structure at the end of learning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_13512 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Dyson Brownian motion and random matrix dynamics of weight matrices during learning Aarts, Gert Hajizadeh, Ouraman Lucini, Biagio Park, Chanju Disordered Systems and Neural Networks Machine Learning High Energy Physics - Lattice During training, weight matrices in machine learning architectures are updated using stochastic gradient descent or variations thereof. In this contribution we employ concepts of random matrix theory to analyse the resulting stochastic matrix dynamics. We first demonstrate that the dynamics can generically be described using Dyson Brownian motion, leading to e.g. eigenvalue repulsion. The level of stochasticity is shown to depend on the ratio of the learning rate and the mini-batch size, explaining the empirically observed linear scaling rule. We verify this linear scaling in the restricted Boltzmann machine. Subsequently we study weight matrix dynamics in transformers (a nano-GPT), following the evolution from a Marchenko-Pastur distribution for eigenvalues at initialisation to a combination with additional structure at the end of learning. |
| title | Dyson Brownian motion and random matrix dynamics of weight matrices during learning |
| topic | Disordered Systems and Neural Networks Machine Learning High Energy Physics - Lattice |
| url | https://arxiv.org/abs/2411.13512 |