Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2407.02028 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916309435940864 |
|---|---|
| author | Li, Xiang Tang, Haoran Chen, Siyu Wang, Ziwei Chen, Ryan Abram, Marcin |
| author_facet | Li, Xiang Tang, Haoran Chen, Siyu Wang, Ziwei Chen, Ryan Abram, Marcin |
| contents | We measure the performance of in-context learning as a function of task novelty and difficulty for open and closed questions. For that purpose, we created a novel benchmark consisting of hard scientific questions, each paired with a context of various relevancy. We show that counter-intuitively, a context that is more aligned with the topic does not always help more than a less relevant context. This effect is especially visible for open questions and questions of high difficulty or novelty. This result reveals a fundamental difference between the treatment of close-form and open-form questions by large-language models and shows a need for a more robust evaluation of in-context learning on the variety of different types of questions. It also poses a new question of how to optimally select a context for large language models, especially in the context of Retrieval Augmented Generation (RAG) systems. Our results suggest that the answer to this question can be highly application-dependent and might be contingent on factors including the format of the question, the perceived difficulty level of the questions, and the novelty or popularity of the information we seek. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_02028 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Why does in-context learning fail sometimes? Evaluating in-context learning on open and closed questions Li, Xiang Tang, Haoran Chen, Siyu Wang, Ziwei Chen, Ryan Abram, Marcin Computation and Language Artificial Intelligence Information Retrieval Machine Learning We measure the performance of in-context learning as a function of task novelty and difficulty for open and closed questions. For that purpose, we created a novel benchmark consisting of hard scientific questions, each paired with a context of various relevancy. We show that counter-intuitively, a context that is more aligned with the topic does not always help more than a less relevant context. This effect is especially visible for open questions and questions of high difficulty or novelty. This result reveals a fundamental difference between the treatment of close-form and open-form questions by large-language models and shows a need for a more robust evaluation of in-context learning on the variety of different types of questions. It also poses a new question of how to optimally select a context for large language models, especially in the context of Retrieval Augmented Generation (RAG) systems. Our results suggest that the answer to this question can be highly application-dependent and might be contingent on factors including the format of the question, the perceived difficulty level of the questions, and the novelty or popularity of the information we seek. |
| title | Why does in-context learning fail sometimes? Evaluating in-context learning on open and closed questions |
| topic | Computation and Language Artificial Intelligence Information Retrieval Machine Learning |
| url | https://arxiv.org/abs/2407.02028 |