In-Context Learning of Linear Dynamical Systems with Transformers: Approximation Bounds and Depth-Separation
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866918164355350528 |
|---|---|
| author | Cole, Frank Zhao, Yuxuan Lu, Yulong Zhang, Tianhao |
| author_facet | Cole, Frank Zhao, Yuxuan Lu, Yulong Zhang, Tianhao |
| contents | This paper investigates approximation-theoretic aspects of the in-context learning capability of the transformers in representing a family of noisy linear dynamical systems. Our first theoretical result establishes an upper bound on the approximation error of multi-layer transformers with respect to an $L^2$-testing loss uniformly defined across tasks. This result demonstrates that transformers with logarithmic depth can achieve error bounds comparable with those of the least-squares estimator. In contrast, our second result establishes a non-diminishing lower bound on the approximation error for a class of single-layer linear transformers, which suggests a depth-separation phenomenon for transformers in the in-context learning of dynamical systems. Moreover, this second result uncovers a critical distinction in the approximation power of single-layer linear transformers when learning from IID versus non-IID data. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_08136 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | In-Context Learning of Linear Dynamical Systems with Transformers: Approximation Bounds and Depth-Separation Cole, Frank Zhao, Yuxuan Lu, Yulong Zhang, Tianhao Machine Learning This paper investigates approximation-theoretic aspects of the in-context learning capability of the transformers in representing a family of noisy linear dynamical systems. Our first theoretical result establishes an upper bound on the approximation error of multi-layer transformers with respect to an $L^2$-testing loss uniformly defined across tasks. This result demonstrates that transformers with logarithmic depth can achieve error bounds comparable with those of the least-squares estimator. In contrast, our second result establishes a non-diminishing lower bound on the approximation error for a class of single-layer linear transformers, which suggests a depth-separation phenomenon for transformers in the in-context learning of dynamical systems. Moreover, this second result uncovers a critical distinction in the approximation power of single-layer linear transformers when learning from IID versus non-IID data. |
| title | In-Context Learning of Linear Dynamical Systems with Transformers: Approximation Bounds and Depth-Separation |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2502.08136 |