Transformers Handle Endogeneity in In-Context Linear Regression
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866908358421774336 |
|---|---|
| author | Liang, Haodong Balasubramanian, Krishnakumar Lai, Lifeng |
| author_facet | Liang, Haodong Balasubramanian, Krishnakumar Lai, Lifeng |
| contents | We explore the capability of transformers to address endogeneity in in-context linear regression. Our main finding is that transformers inherently possess a mechanism to handle endogeneity effectively using instrumental variables (IV). First, we demonstrate that the transformer architecture can emulate a gradient-based bi-level optimization procedure that converges to the widely used two-stage least squares $(\textsf{2SLS})$ solution at an exponential rate. Next, we propose an in-context pretraining scheme and provide theoretical guarantees showing that the global minimizer of the pre-training loss achieves a small excess loss. Our extensive experiments validate these theoretical findings, showing that the trained transformer provides more robust and reliable in-context predictions and coefficient estimates than the $\textsf{2SLS}$ method, in the presence of endogeneity. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_01265 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Transformers Handle Endogeneity in In-Context Linear Regression Liang, Haodong Balasubramanian, Krishnakumar Lai, Lifeng Machine Learning Artificial Intelligence Econometrics Statistics Theory We explore the capability of transformers to address endogeneity in in-context linear regression. Our main finding is that transformers inherently possess a mechanism to handle endogeneity effectively using instrumental variables (IV). First, we demonstrate that the transformer architecture can emulate a gradient-based bi-level optimization procedure that converges to the widely used two-stage least squares $(\textsf{2SLS})$ solution at an exponential rate. Next, we propose an in-context pretraining scheme and provide theoretical guarantees showing that the global minimizer of the pre-training loss achieves a small excess loss. Our extensive experiments validate these theoretical findings, showing that the trained transformer provides more robust and reliable in-context predictions and coefficient estimates than the $\textsf{2SLS}$ method, in the presence of endogeneity. |
| title | Transformers Handle Endogeneity in In-Context Linear Regression |
| topic | Machine Learning Artificial Intelligence Econometrics Statistics Theory |
| url | https://arxiv.org/abs/2410.01265 |