Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2602.17171 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908840616787968 |
|---|---|
| author | Goel, Ayush Kohli, Arjun Somvanshi, Sarvagya |
| author_facet | Goel, Ayush Kohli, Arjun Somvanshi, Sarvagya |
| contents | Recent work has demonstrated that transformers and linear attention models can perform in-context learning (ICL) on simple function classes, such as linear regression. In this paper, we empirically study how these two attention mechanisms differ in their ICL behavior on the canonical linear-regression task of Garg et al. We evaluate learning quality (MSE), convergence, and generalization behavior of each architecture. We also analyze how increasing model depth affects ICL performance. Our results illustrate both the similarities and limitations of linear attention relative to quadratic attention in this setting. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_17171 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | In-Context Learning in Linear vs. Quadratic Attention Models: An Empirical Study on Regression Tasks Goel, Ayush Kohli, Arjun Somvanshi, Sarvagya Machine Learning Artificial Intelligence Recent work has demonstrated that transformers and linear attention models can perform in-context learning (ICL) on simple function classes, such as linear regression. In this paper, we empirically study how these two attention mechanisms differ in their ICL behavior on the canonical linear-regression task of Garg et al. We evaluate learning quality (MSE), convergence, and generalization behavior of each architecture. We also analyze how increasing model depth affects ICL performance. Our results illustrate both the similarities and limitations of linear attention relative to quadratic attention in this setting. |
| title | In-Context Learning in Linear vs. Quadratic Attention Models: An Empirical Study on Regression Tasks |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2602.17171 |