Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2512.15934 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910005268054016 |
|---|---|
| author | Fan, Jiashuo Rosu, Paul Wang, Aaron T. Li, Zeyu Michael Carin, Lawrence Cheng, Xiang |
| author_facet | Fan, Jiashuo Rosu, Paul Wang, Aaron T. Li, Zeyu Michael Carin, Lawrence Cheng, Xiang |
| contents | There has been significant recent interest in understanding the capacity of Transformers for in-context learning (ICL), yet most theory focuses on supervised settings with explicitly labeled pairs. In practice, Transformers often perform well even when labels are sparse or absent, suggesting crucial structure within unlabeled contextual demonstrations. We introduce and study in-context semi-supervised learning (IC-SSL), where a small set of labeled examples is accompanied by many unlabeled points, and show that Transformers can leverage the unlabeled context to learn a robust, context-dependent representation. This representation enables accurate predictions and markedly improves performance in low-label regimes, offering foundational insights into how Transformers exploit unlabeled context for representation learning within the ICL framework. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_15934 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | In-Context Semi-Supervised Learning Fan, Jiashuo Rosu, Paul Wang, Aaron T. Li, Zeyu Michael Carin, Lawrence Cheng, Xiang Machine Learning There has been significant recent interest in understanding the capacity of Transformers for in-context learning (ICL), yet most theory focuses on supervised settings with explicitly labeled pairs. In practice, Transformers often perform well even when labels are sparse or absent, suggesting crucial structure within unlabeled contextual demonstrations. We introduce and study in-context semi-supervised learning (IC-SSL), where a small set of labeled examples is accompanied by many unlabeled points, and show that Transformers can leverage the unlabeled context to learn a robust, context-dependent representation. This representation enables accurate predictions and markedly improves performance in low-label regimes, offering foundational insights into how Transformers exploit unlabeled context for representation learning within the ICL framework. |
| title | In-Context Semi-Supervised Learning |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2512.15934 |