Dynamical Behaviors of the Gradient Flows for In-Context Learning
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916538257244160 |
|---|---|
| author | Lu, Songtao Lu, Yingdong Nowicki, Tomasz |
| author_facet | Lu, Songtao Lu, Yingdong Nowicki, Tomasz |
| contents | We derive the system of differential equations for the gradient flow characterizing the training process of linear in-context learning in full generality. Next, we explore the geometric structure of the gradient flows in two instances, including identifying its invariants, optimum, and saddle points. This understanding allows us to quantify the behavior of the two gradient flows under the full generality of parameters and data. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_16683 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Dynamical Behaviors of the Gradient Flows for In-Context Learning Lu, Songtao Lu, Yingdong Nowicki, Tomasz Dynamical Systems We derive the system of differential equations for the gradient flow characterizing the training process of linear in-context learning in full generality. Next, we explore the geometric structure of the gradient flows in two instances, including identifying its invariants, optimum, and saddle points. This understanding allows us to quantify the behavior of the two gradient flows under the full generality of parameters and data. |
| title | Dynamical Behaviors of the Gradient Flows for In-Context Learning |
| topic | Dynamical Systems |
| url | https://arxiv.org/abs/2412.16683 |