Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2605.18512 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917508206821376 |
|---|---|
| author | Wang, Haochun Yang, Chaofen Liu, Jiatong Wang, Jingbo Qiang, Zewen Zhao, Sendong Qin, Bing Liu, Ting |
| author_facet | Wang, Haochun Yang, Chaofen Liu, Jiatong Wang, Jingbo Qiang, Zewen Zhao, Sendong Qin, Bing Liu, Ting |
| contents | In-context learning (ICL) is highly sensitive to which demonstrations appear in the prompt, but selecting them is expensive because the space of possible demonstration contexts and combinations is enormous. We argue that demonstration selection is \emph{easier to judge than to find}: predicting whether a specific query--context pair $(q,D)$ will succeed is cheaper and more general than searching for an optimal $D^\star$. Based on this insight, we propose DiSP, a sample-and-judge framework that stratifies queries by difficulty. DiSP runs random demonstration trials to estimate success rate of each training query, trains a lightweight router to predict difficulty from the query, and trains level-specific judges for sampled demonstrations. At inference, DiSP performs stop-on-acceptance judging under an explicit budget, emitting diagnostic risk tags when no suitable context is found. Across five classification datasets with Llama~3--8B and Qwen~2.5--7B, DiSP achieves the best average accuracy, improving over strong learned selection baselines by up to 3.4\%, while achieving up to $23\times$ end-to-end wall-clock speedup. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_18512 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Easier to Judge than to Find: Predicting In-Context Learning Success for Demonstration Selection Wang, Haochun Yang, Chaofen Liu, Jiatong Wang, Jingbo Qiang, Zewen Zhao, Sendong Qin, Bing Liu, Ting Computation and Language In-context learning (ICL) is highly sensitive to which demonstrations appear in the prompt, but selecting them is expensive because the space of possible demonstration contexts and combinations is enormous. We argue that demonstration selection is \emph{easier to judge than to find}: predicting whether a specific query--context pair $(q,D)$ will succeed is cheaper and more general than searching for an optimal $D^\star$. Based on this insight, we propose DiSP, a sample-and-judge framework that stratifies queries by difficulty. DiSP runs random demonstration trials to estimate success rate of each training query, trains a lightweight router to predict difficulty from the query, and trains level-specific judges for sampled demonstrations. At inference, DiSP performs stop-on-acceptance judging under an explicit budget, emitting diagnostic risk tags when no suitable context is found. Across five classification datasets with Llama~3--8B and Qwen~2.5--7B, DiSP achieves the best average accuracy, improving over strong learned selection baselines by up to 3.4\%, while achieving up to $23\times$ end-to-end wall-clock speedup. |
| title | Easier to Judge than to Find: Predicting In-Context Learning Success for Demonstration Selection |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2605.18512 |