Efficient and Minimax Optimal In-context Nonparametric Regression with Transformers
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911698003165184 |
|---|---|
| author | Ching, Michelle Popescu, Ioana Smith, Nico Ma, Tianyi Underwood, William G. Samworth, Richard J. |
| author_facet | Ching, Michelle Popescu, Ioana Smith, Nico Ma, Tianyi Underwood, William G. Samworth, Richard J. |
| contents | We study in-context learning for nonparametric regression with $α$-Hölder smooth regression functions, for some $α>0$. We prove that, with $n$ in-context examples and $d$-dimensional regression covariates, a pretrained transformer with $Θ(\log n)$ parameters and $Ω\bigl(n^{2α/(2α+d)}\log^3 n\bigr)$ pretraining sequences can achieve the minimax optimal rate of convergence $O\bigl(n^{-2α/(2α+d)}\bigr)$ in mean squared error. Our result requires substantially fewer transformer parameters and pretraining sequences than previous results in the literature. This is achieved by showing that transformers are able to approximate local polynomial estimators efficiently by implementing a kernel-weighted polynomial basis and then running gradient descent. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_15014 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Efficient and Minimax Optimal In-context Nonparametric Regression with Transformers Ching, Michelle Popescu, Ioana Smith, Nico Ma, Tianyi Underwood, William G. Samworth, Richard J. Machine Learning Statistics Theory 62G08, 68T07 We study in-context learning for nonparametric regression with $α$-Hölder smooth regression functions, for some $α>0$. We prove that, with $n$ in-context examples and $d$-dimensional regression covariates, a pretrained transformer with $Θ(\log n)$ parameters and $Ω\bigl(n^{2α/(2α+d)}\log^3 n\bigr)$ pretraining sequences can achieve the minimax optimal rate of convergence $O\bigl(n^{-2α/(2α+d)}\bigr)$ in mean squared error. Our result requires substantially fewer transformer parameters and pretraining sequences than previous results in the literature. This is achieved by showing that transformers are able to approximate local polynomial estimators efficiently by implementing a kernel-weighted polynomial basis and then running gradient descent. |
| title | Efficient and Minimax Optimal In-context Nonparametric Regression with Transformers |
| topic | Machine Learning Statistics Theory 62G08, 68T07 |
| url | https://arxiv.org/abs/2601.15014 |