On the Training Convergence of Transformers for In-Context Classification of Gaussian Mixtures
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912401527406592 |
|---|---|
| author | Shen, Wei Zhou, Ruida Yang, Jing Shen, Cong |
| author_facet | Shen, Wei Zhou, Ruida Yang, Jing Shen, Cong |
| contents | Although transformers have demonstrated impressive capabilities for in-context learning (ICL) in practice, theoretical understanding of the underlying mechanism that allows transformers to perform ICL is still in its infancy. This work aims to theoretically study the training dynamics of transformers for in-context classification tasks. We demonstrate that, for in-context classification of Gaussian mixtures under certain assumptions, a single-layer transformer trained via gradient descent converges to a globally optimal model at a linear rate. We further quantify the impact of the training and testing prompt lengths on the ICL inference error of the trained transformer. We show that when the lengths of training and testing prompts are sufficiently large, the prediction of the trained transformer approaches the ground truth distribution of the labels. Experimental results corroborate the theoretical findings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_11778 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | On the Training Convergence of Transformers for In-Context Classification of Gaussian Mixtures Shen, Wei Zhou, Ruida Yang, Jing Shen, Cong Machine Learning Information Theory Although transformers have demonstrated impressive capabilities for in-context learning (ICL) in practice, theoretical understanding of the underlying mechanism that allows transformers to perform ICL is still in its infancy. This work aims to theoretically study the training dynamics of transformers for in-context classification tasks. We demonstrate that, for in-context classification of Gaussian mixtures under certain assumptions, a single-layer transformer trained via gradient descent converges to a globally optimal model at a linear rate. We further quantify the impact of the training and testing prompt lengths on the ICL inference error of the trained transformer. We show that when the lengths of training and testing prompts are sufficiently large, the prediction of the trained transformer approaches the ground truth distribution of the labels. Experimental results corroborate the theoretical findings. |
| title | On the Training Convergence of Transformers for In-Context Classification of Gaussian Mixtures |
| topic | Machine Learning Information Theory |
| url | https://arxiv.org/abs/2410.11778 |