Exploring Imbalanced Annotations for Effective In-Context Learning
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913866360815616 |
|---|---|
| author | Gao, Hongfu Zhang, Feipeng Zeng, Hao Meng, Deyu Jing, Bingyi Wei, Hongxin |
| author_facet | Gao, Hongfu Zhang, Feipeng Zeng, Hao Meng, Deyu Jing, Bingyi Wei, Hongxin |
| contents | Large language models (LLMs) have shown impressive performance on downstream tasks through in-context learning (ICL), which heavily relies on the demonstrations selected from annotated datasets. However, these datasets often exhibit long-tailed class distributions in real-world scenarios, leading to biased demonstration selection. In this work, we show that such class imbalances significantly degrade the ICL performance across various tasks, regardless of selection methods. Moreover, classical rebalancing methods, which focus solely on class weights, yield poor performance due to neglecting condition bias--skewed feature distributions within classes. To address this, we propose Reweighting with Conditional Bias (dubbed RCB), a simple and complementary approach to enhance ICL performance under class imbalance. In particular, RCB estimates conditional bias using a balanced subset and re-weights demonstration scores based on both class weight and conditional bias. In effect, RCB prevents over-selection from dominant classes while preserving the efficacy of current selection methods. Extensive experiments on common benchmarks demonstrate the effectiveness of our method, improving the average accuracy of current selection methods by up to 5.42%. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_04037 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Exploring Imbalanced Annotations for Effective In-Context Learning Gao, Hongfu Zhang, Feipeng Zeng, Hao Meng, Deyu Jing, Bingyi Wei, Hongxin Computation and Language Machine Learning Large language models (LLMs) have shown impressive performance on downstream tasks through in-context learning (ICL), which heavily relies on the demonstrations selected from annotated datasets. However, these datasets often exhibit long-tailed class distributions in real-world scenarios, leading to biased demonstration selection. In this work, we show that such class imbalances significantly degrade the ICL performance across various tasks, regardless of selection methods. Moreover, classical rebalancing methods, which focus solely on class weights, yield poor performance due to neglecting condition bias--skewed feature distributions within classes. To address this, we propose Reweighting with Conditional Bias (dubbed RCB), a simple and complementary approach to enhance ICL performance under class imbalance. In particular, RCB estimates conditional bias using a balanced subset and re-weights demonstration scores based on both class weight and conditional bias. In effect, RCB prevents over-selection from dominant classes while preserving the efficacy of current selection methods. Extensive experiments on common benchmarks demonstrate the effectiveness of our method, improving the average accuracy of current selection methods by up to 5.42%. |
| title | Exploring Imbalanced Annotations for Effective In-Context Learning |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2502.04037 |