Convergent Stochastic Training of Attention and Understanding LoRA
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914545155440640 |
|---|---|
| author | Sun, Zhengkai Kumar, Dibyakanti Frangi, Alejandro F Mukherjee, Anirbit Sun, Mingfei |
| author_facet | Sun, Zhengkai Kumar, Dibyakanti Frangi, Alejandro F Mukherjee, Anirbit Sun, Mingfei |
| contents | Transformers have revolutionized machine learning and deploying attention layers in the model is increasingly standard across a myriad of applications. Further, for large models, it is common to implement Low Rank Adaptation (LoRA), whereby a factorized parameterization of them is trained, to achieve a surprisingly beneficial accuracy-size trade-off. In this work, via a unified framework we rigorously establish trainability of such models under stochastic methods. We prove that for any mild regularization, the empirical regression loss on a attention layer and LoRA on a shallow neural net, both induce Poincaré inequality for the corresponding Gibbs' measure. Then it follows via invoking recent results that a certain SDE, which mimics the SGD, minimizes the corresponding losses. In both the cases, our first-of-its-kind results of trainability on attention and nets, do not rely on any assumptions on the data or the size of the architecture. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_07959 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Convergent Stochastic Training of Attention and Understanding LoRA Sun, Zhengkai Kumar, Dibyakanti Frangi, Alejandro F Mukherjee, Anirbit Sun, Mingfei Machine Learning Functional Analysis Probability Transformers have revolutionized machine learning and deploying attention layers in the model is increasingly standard across a myriad of applications. Further, for large models, it is common to implement Low Rank Adaptation (LoRA), whereby a factorized parameterization of them is trained, to achieve a surprisingly beneficial accuracy-size trade-off. In this work, via a unified framework we rigorously establish trainability of such models under stochastic methods. We prove that for any mild regularization, the empirical regression loss on a attention layer and LoRA on a shallow neural net, both induce Poincaré inequality for the corresponding Gibbs' measure. Then it follows via invoking recent results that a certain SDE, which mimics the SGD, minimizes the corresponding losses. In both the cases, our first-of-its-kind results of trainability on attention and nets, do not rely on any assumptions on the data or the size of the architecture. |
| title | Convergent Stochastic Training of Attention and Understanding LoRA |
| topic | Machine Learning Functional Analysis Probability |
| url | https://arxiv.org/abs/2605.07959 |