Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2309.07900 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911767188209664 |
|---|---|
| author | Gao, Lingyu Chaudhary, Aditi Srinivasan, Krishna Hashimoto, Kazuma Raman, Karthik Bendersky, Michael |
| author_facet | Gao, Lingyu Chaudhary, Aditi Srinivasan, Krishna Hashimoto, Kazuma Raman, Karthik Bendersky, Michael |
| contents | In-context learning (ICL) i.e. showing LLMs only a few task-specific demonstrations has led to downstream gains with no task-specific fine-tuning required. However, LLMs are sensitive to the choice of prompts, and therefore a crucial research question is how to select good demonstrations for ICL. One effective strategy is leveraging semantic similarity between the ICL demonstrations and test inputs by using a text retriever, which however is sub-optimal as that does not consider the LLM's existing knowledge about that task. From prior work (Lyu et al., 2023), we already know that labels paired with the demonstrations bias the model predictions. This leads us to our hypothesis whether considering LLM's existing knowledge about the task, especially with respect to the output label space can help in a better demonstration selection strategy. Through extensive experimentation on three text classification tasks, we find that it is beneficial to not only choose semantically similar ICL demonstrations but also to choose those demonstrations that help resolve the inherent label ambiguity surrounding the test example. Interestingly, we find that including demonstrations that the LLM previously mis-classified and also fall on the test example's decision boundary, brings the most performance gain. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2309_07900 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Ambiguity-Aware In-Context Learning with Large Language Models Gao, Lingyu Chaudhary, Aditi Srinivasan, Krishna Hashimoto, Kazuma Raman, Karthik Bendersky, Michael Computation and Language Information Retrieval In-context learning (ICL) i.e. showing LLMs only a few task-specific demonstrations has led to downstream gains with no task-specific fine-tuning required. However, LLMs are sensitive to the choice of prompts, and therefore a crucial research question is how to select good demonstrations for ICL. One effective strategy is leveraging semantic similarity between the ICL demonstrations and test inputs by using a text retriever, which however is sub-optimal as that does not consider the LLM's existing knowledge about that task. From prior work (Lyu et al., 2023), we already know that labels paired with the demonstrations bias the model predictions. This leads us to our hypothesis whether considering LLM's existing knowledge about the task, especially with respect to the output label space can help in a better demonstration selection strategy. Through extensive experimentation on three text classification tasks, we find that it is beneficial to not only choose semantically similar ICL demonstrations but also to choose those demonstrations that help resolve the inherent label ambiguity surrounding the test example. Interestingly, we find that including demonstrations that the LLM previously mis-classified and also fall on the test example's decision boundary, brings the most performance gain. |
| title | Ambiguity-Aware In-Context Learning with Large Language Models |
| topic | Computation and Language Information Retrieval |
| url | https://arxiv.org/abs/2309.07900 |