In-Context Learning Distillation for Efficient Few-Shot Fine-Tuning
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913616569040896 |
|---|---|
| author | Duan, Yifei Li, Liu Zhai, Zirui Yao, Jinxia |
| author_facet | Duan, Yifei Li, Liu Zhai, Zirui Yao, Jinxia |
| contents | We applied few-shot in-context learning on the OPT-1.3B model for the natural language inference task and employed knowledge distillation to internalize the context information, reducing model parameter from 1.3B to 125M and achieving a size reduction from 2.5GB to 0.25GB. Compared to using in-context learning alone on similarly sized models, this context distillation approach achieved a nearly 50% improvement in out-of-domain accuracy, demonstrating superior knowledge transfer capabilities over prompt-based methods. Furthermore, this approach reduced memory consumption by up to 60% while delivering a 20% improvement in out-of-domain accuracy compared to conventional pattern-based fine-tuning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_13243 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | In-Context Learning Distillation for Efficient Few-Shot Fine-Tuning Duan, Yifei Li, Liu Zhai, Zirui Yao, Jinxia Computation and Language We applied few-shot in-context learning on the OPT-1.3B model for the natural language inference task and employed knowledge distillation to internalize the context information, reducing model parameter from 1.3B to 125M and achieving a size reduction from 2.5GB to 0.25GB. Compared to using in-context learning alone on similarly sized models, this context distillation approach achieved a nearly 50% improvement in out-of-domain accuracy, demonstrating superior knowledge transfer capabilities over prompt-based methods. Furthermore, this approach reduced memory consumption by up to 60% while delivering a 20% improvement in out-of-domain accuracy compared to conventional pattern-based fine-tuning. |
| title | In-Context Learning Distillation for Efficient Few-Shot Fine-Tuning |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2412.13243 |