In-Context Learning of Linear Systems: Generalization Theory and Applications to Operator Learning
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866912390717636608 |
|---|---|
| author | Cole, Frank Lu, Yulong Xu, Wuzhe Zhang, Tianhao |
| author_facet | Cole, Frank Lu, Yulong Xu, Wuzhe Zhang, Tianhao |
| contents | We study theoretical guarantees for solving linear systems in-context using a linear transformer architecture. For in-domain generalization, we provide neural scaling laws that bound the generalization error in terms of the number of tasks and sizes of samples used in training and inference. For out-of-domain generalization, we find that the behavior of trained transformers under task distribution shifts depends crucially on the distribution of the tasks seen during training. We introduce a novel notion of task diversity and show that it defines a necessary and sufficient condition for pre-trained transformers generalize under task distribution shifts. We also explore applications of learning linear systems in-context, such as to in-context operator learning for PDEs. Finally, we provide some numerical experiments to validate the established theory. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_12293 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | In-Context Learning of Linear Systems: Generalization Theory and Applications to Operator Learning Cole, Frank Lu, Yulong Xu, Wuzhe Zhang, Tianhao Machine Learning Numerical Analysis We study theoretical guarantees for solving linear systems in-context using a linear transformer architecture. For in-domain generalization, we provide neural scaling laws that bound the generalization error in terms of the number of tasks and sizes of samples used in training and inference. For out-of-domain generalization, we find that the behavior of trained transformers under task distribution shifts depends crucially on the distribution of the tasks seen during training. We introduce a novel notion of task diversity and show that it defines a necessary and sufficient condition for pre-trained transformers generalize under task distribution shifts. We also explore applications of learning linear systems in-context, such as to in-context operator learning for PDEs. Finally, we provide some numerical experiments to validate the established theory. |
| title | In-Context Learning of Linear Systems: Generalization Theory and Applications to Operator Learning |
| topic | Machine Learning Numerical Analysis |
| url | https://arxiv.org/abs/2409.12293 |