Revisiting Demonstration Selection Strategies in In-Context Learning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913402267369472 |
|---|---|
| author | Peng, Keqin Ding, Liang Yuan, Yancheng Liu, Xuebo Zhang, Min Ouyang, Yuanxin Tao, Dacheng |
| author_facet | Peng, Keqin Ding, Liang Yuan, Yancheng Liu, Xuebo Zhang, Min Ouyang, Yuanxin Tao, Dacheng |
| contents | Large language models (LLMs) have shown an impressive ability to perform a wide range of tasks using in-context learning (ICL), where a few examples are used to describe a task to the model. However, the performance of ICL varies significantly with the choice of demonstrations, and it is still unclear why this happens or what factors will influence its choice. In this work, we first revisit the factors contributing to this variance from both data and model aspects, and find that the choice of demonstration is both data- and model-dependent. We further proposed a data- and model-dependent demonstration selection method, \textbf{TopK + ConE}, based on the assumption that \textit{the performance of a demonstration positively correlates with its contribution to the model's understanding of the test samples}, resulting in a simple and effective recipe for ICL. Empirically, our method yields consistent improvements in both language understanding and generation tasks with different model scales. Further analyses confirm that, besides the generality and stability under different circumstances, our method provides a unified explanation for the effectiveness of previous methods. Code will be released. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_12087 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Revisiting Demonstration Selection Strategies in In-Context Learning Peng, Keqin Ding, Liang Yuan, Yancheng Liu, Xuebo Zhang, Min Ouyang, Yuanxin Tao, Dacheng Computation and Language Large language models (LLMs) have shown an impressive ability to perform a wide range of tasks using in-context learning (ICL), where a few examples are used to describe a task to the model. However, the performance of ICL varies significantly with the choice of demonstrations, and it is still unclear why this happens or what factors will influence its choice. In this work, we first revisit the factors contributing to this variance from both data and model aspects, and find that the choice of demonstration is both data- and model-dependent. We further proposed a data- and model-dependent demonstration selection method, \textbf{TopK + ConE}, based on the assumption that \textit{the performance of a demonstration positively correlates with its contribution to the model's understanding of the test samples}, resulting in a simple and effective recipe for ICL. Empirically, our method yields consistent improvements in both language understanding and generation tasks with different model scales. Further analyses confirm that, besides the generality and stability under different circumstances, our method provides a unified explanation for the effectiveness of previous methods. Code will be released. |
| title | Revisiting Demonstration Selection Strategies in In-Context Learning |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2401.12087 |