Scaling Laws for Many-Shot In-Context Learning with Self-Generated Annotations
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910958436220928 |
|---|---|
| author | Gu, Zhengyao Zou, Henry Peng Chen, Yankai Liu, Aiwei Zhang, Weizhi Yu, Philip S. |
| author_facet | Gu, Zhengyao Zou, Henry Peng Chen, Yankai Liu, Aiwei Zhang, Weizhi Yu, Philip S. |
| contents | The high cost of obtaining high-quality annotated data for in-context learning (ICL) has motivated the development of methods that use self-generated annotations in place of ground-truth labels. While these approaches have shown promising results in few-shot settings, they generally do not scale to many-shot scenarios. In this work, we study ICL with self-generated examples using a framework analogous to traditional semi-supervised learning, consisting of annotation generation, demonstration selection, and in-context inference. Within this framework, we propose a simple baseline that outperforms ground-truth ICL in zero-shot, few-shot, and many-shot settings. Notably, we observe a scaling law with this baseline, where optimal performance is achieved with more than 1,000 demonstrations. To fully exploit the many-shot capabilities of semi-supervised ICL, we introduce IterPSD, an iterative annotation approach that integrates iterative refinement and curriculum pseudo-labeling techniques from semi-supervised learning, yielding up to 6.8% additional gains on classification tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_03062 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Scaling Laws for Many-Shot In-Context Learning with Self-Generated Annotations Gu, Zhengyao Zou, Henry Peng Chen, Yankai Liu, Aiwei Zhang, Weizhi Yu, Philip S. Computation and Language Artificial Intelligence Information Retrieval Machine Learning The high cost of obtaining high-quality annotated data for in-context learning (ICL) has motivated the development of methods that use self-generated annotations in place of ground-truth labels. While these approaches have shown promising results in few-shot settings, they generally do not scale to many-shot scenarios. In this work, we study ICL with self-generated examples using a framework analogous to traditional semi-supervised learning, consisting of annotation generation, demonstration selection, and in-context inference. Within this framework, we propose a simple baseline that outperforms ground-truth ICL in zero-shot, few-shot, and many-shot settings. Notably, we observe a scaling law with this baseline, where optimal performance is achieved with more than 1,000 demonstrations. To fully exploit the many-shot capabilities of semi-supervised ICL, we introduce IterPSD, an iterative annotation approach that integrates iterative refinement and curriculum pseudo-labeling techniques from semi-supervised learning, yielding up to 6.8% additional gains on classification tasks. |
| title | Scaling Laws for Many-Shot In-Context Learning with Self-Generated Annotations |
| topic | Computation and Language Artificial Intelligence Information Retrieval Machine Learning |
| url | https://arxiv.org/abs/2503.03062 |