Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2406.13274 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929688429985792 |
|---|---|
| author | Berger, Uri Baumel, Tal Stanovsky, Gabriel |
| author_facet | Berger, Uri Baumel, Tal Stanovsky, Gabriel |
| contents | Few shot in-context learning (ICL) typically assumes access to large annotated training sets. However, in many real world scenarios, such as domain adaptation, there is only a limited budget to annotate a small number of samples, with the goal of maximizing downstream performance. We study various methods for selecting samples to annotate within a predefined budget, focusing on token classification tasks, which are expensive to annotate and are relatively less studied in ICL setups. Across various tasks, models, and datasets, we observe that no method significantly outperforms the others, with most yielding similar results, including random sample selection for annotation. Moreover, we demonstrate that a relatively small annotated sample pool can achieve performance comparable to using the entire training set. We hope that future work adopts our realistic paradigm which takes annotation budget into account. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_13274 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | In-Context Learning on a Budget: A Case Study in Token Classification Berger, Uri Baumel, Tal Stanovsky, Gabriel Computation and Language Few shot in-context learning (ICL) typically assumes access to large annotated training sets. However, in many real world scenarios, such as domain adaptation, there is only a limited budget to annotate a small number of samples, with the goal of maximizing downstream performance. We study various methods for selecting samples to annotate within a predefined budget, focusing on token classification tasks, which are expensive to annotate and are relatively less studied in ICL setups. Across various tasks, models, and datasets, we observe that no method significantly outperforms the others, with most yielding similar results, including random sample selection for annotation. Moreover, we demonstrate that a relatively small annotated sample pool can achieve performance comparable to using the entire training set. We hope that future work adopts our realistic paradigm which takes annotation budget into account. |
| title | In-Context Learning on a Budget: A Case Study in Token Classification |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2406.13274 |