OptiSeq: Ordering Examples On-The-Fly for In-Context Learning
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917928393244672 |
|---|---|
| author | Bhope, Rahul Atul Venkateswaran, Praveen Jayaram, K. R. Isahagian, Vatche Muthusamy, Vinod Venkatasubramanian, Nalini |
| author_facet | Bhope, Rahul Atul Venkateswaran, Praveen Jayaram, K. R. Isahagian, Vatche Muthusamy, Vinod Venkatasubramanian, Nalini |
| contents | Developers using LLMs and LLM-based agents in their applications have provided plenty of anecdotal evidence that in-context-learning (ICL) is fragile. In this paper, we show that in addition to the quantity and quality of examples, the order in which the in-context examples are listed in the prompt affects the output of the LLM and, consequently, their performance. While prior work has explored improving ICL through dataset-dependent techniques, we introduce OptiSeq, a purely inference-time, dataset-free optimization method that efficiently determines the best example order. OptiSeq leverages log probabilities of LLM-generated outputs to systematically prune the search space of possible orderings and recommend the best order(s) by distinguishing orderings that yield high levels of accuracy and those that underperform. Extensive empirical evaluation on multiple LLMs, datasets, and prompts demonstrate that OptiSeq improves accuracy by 5.5 - 10.5 percentage points across multiple tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_15030 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | OptiSeq: Ordering Examples On-The-Fly for In-Context Learning Bhope, Rahul Atul Venkateswaran, Praveen Jayaram, K. R. Isahagian, Vatche Muthusamy, Vinod Venkatasubramanian, Nalini Machine Learning Artificial Intelligence Computation and Language Performance Developers using LLMs and LLM-based agents in their applications have provided plenty of anecdotal evidence that in-context-learning (ICL) is fragile. In this paper, we show that in addition to the quantity and quality of examples, the order in which the in-context examples are listed in the prompt affects the output of the LLM and, consequently, their performance. While prior work has explored improving ICL through dataset-dependent techniques, we introduce OptiSeq, a purely inference-time, dataset-free optimization method that efficiently determines the best example order. OptiSeq leverages log probabilities of LLM-generated outputs to systematically prune the search space of possible orderings and recommend the best order(s) by distinguishing orderings that yield high levels of accuracy and those that underperform. Extensive empirical evaluation on multiple LLMs, datasets, and prompts demonstrate that OptiSeq improves accuracy by 5.5 - 10.5 percentage points across multiple tasks. |
| title | OptiSeq: Ordering Examples On-The-Fly for In-Context Learning |
| topic | Machine Learning Artificial Intelligence Computation and Language Performance |
| url | https://arxiv.org/abs/2501.15030 |