NoisyICL: A Little Noise in Model Parameters Calibrates In-context Learning
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914679603855360 |
|---|---|
| author | Zhao, Yufeng Sakai, Yoshihiro Inoue, Naoya |
| author_facet | Zhao, Yufeng Sakai, Yoshihiro Inoue, Naoya |
| contents | In-Context Learning (ICL) is suffering from unsatisfactory performance and under-calibration due to high prior bias and unfaithful confidence. Some previous works fine-tuned language models for better ICL performance with enormous datasets and computing costs. In this paper, we propose NoisyICL, simply perturbing the model parameters by random noises to strive for better performance and calibration. Our experiments on two models and 12 downstream datasets show that NoisyICL can help ICL produce more accurate predictions. Our further analysis indicates that NoisyICL enables the model to provide more fair predictions, and also with more faithful confidence. Therefore, we believe that NoisyICL is an effective calibration of ICL. Our experimental code is uploaded to Github. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_05515 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | NoisyICL: A Little Noise in Model Parameters Calibrates In-context Learning Zhao, Yufeng Sakai, Yoshihiro Inoue, Naoya Computation and Language Artificial Intelligence In-Context Learning (ICL) is suffering from unsatisfactory performance and under-calibration due to high prior bias and unfaithful confidence. Some previous works fine-tuned language models for better ICL performance with enormous datasets and computing costs. In this paper, we propose NoisyICL, simply perturbing the model parameters by random noises to strive for better performance and calibration. Our experiments on two models and 12 downstream datasets show that NoisyICL can help ICL produce more accurate predictions. Our further analysis indicates that NoisyICL enables the model to provide more fair predictions, and also with more faithful confidence. Therefore, we believe that NoisyICL is an effective calibration of ICL. Our experimental code is uploaded to Github. |
| title | NoisyICL: A Little Noise in Model Parameters Calibrates In-context Learning |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2402.05515 |