Guardado en:
| Autores principales: | , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | https://arxiv.org/abs/2410.01548 |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866929570966405120 |
|---|---|
| author | Wang, Dingzirui Zhang, Xuanliang Chen, Qiguang Dou, Longxu Xu, Xiao Cao, Rongyu Ma, Yingwei Zhu, Qingfu Che, Wanxiang Li, Binhua Huang, Fei Li, Yongbin |
| author_facet | Wang, Dingzirui Zhang, Xuanliang Chen, Qiguang Dou, Longxu Xu, Xiao Cao, Rongyu Ma, Yingwei Zhu, Qingfu Che, Wanxiang Li, Binhua Huang, Fei Li, Yongbin |
| contents | In-context learning (ICL) is an effective approach to help large language models (LLMs) adapt to various tasks by providing demonstrations of the target task. Considering the high cost of labeling demonstrations, many methods propose synthesizing demonstrations from scratch using LLMs. However, the quality of the demonstrations synthesized from scratch is limited by the capabilities and knowledge of LLMs. To address this, inspired by transfer learning, we propose In-Context Transfer Learning (ICTL), which synthesizes target task demonstrations by transferring labeled demonstrations from similar source tasks. ICTL consists of two steps: source sampling and target transfer. First, we define an optimization objective, which minimizes transfer error to sample source demonstrations similar to the target task. Then, we employ LLMs to transfer the sampled source demonstrations to the target task, matching the definition and format of the target task. Experiments on Super-NI show that ICTL outperforms synthesis from scratch by 2.0% on average, demonstrating the effectiveness of our method. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_01548 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | In-Context Transfer Learning: Demonstration Synthesis by Transferring Similar Tasks Wang, Dingzirui Zhang, Xuanliang Chen, Qiguang Dou, Longxu Xu, Xiao Cao, Rongyu Ma, Yingwei Zhu, Qingfu Che, Wanxiang Li, Binhua Huang, Fei Li, Yongbin Computation and Language Machine Learning In-context learning (ICL) is an effective approach to help large language models (LLMs) adapt to various tasks by providing demonstrations of the target task. Considering the high cost of labeling demonstrations, many methods propose synthesizing demonstrations from scratch using LLMs. However, the quality of the demonstrations synthesized from scratch is limited by the capabilities and knowledge of LLMs. To address this, inspired by transfer learning, we propose In-Context Transfer Learning (ICTL), which synthesizes target task demonstrations by transferring labeled demonstrations from similar source tasks. ICTL consists of two steps: source sampling and target transfer. First, we define an optimization objective, which minimizes transfer error to sample source demonstrations similar to the target task. Then, we employ LLMs to transfer the sampled source demonstrations to the target task, matching the definition and format of the target task. Experiments on Super-NI show that ICTL outperforms synthesis from scratch by 2.0% on average, demonstrating the effectiveness of our method. |
| title | In-Context Transfer Learning: Demonstration Synthesis by Transferring Similar Tasks |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2410.01548 |