Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2410.02210 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912056646565888 |
|---|---|
| author | Cheng, Wei Wang, Tianlu Ji, Yanmin Yang, Fan Tan, Keren Zheng, Yiyu |
| author_facet | Cheng, Wei Wang, Tianlu Ji, Yanmin Yang, Fan Tan, Keren Zheng, Yiyu |
| contents | While in-context learning with large language models (LLMs) has shown impressive performance, we have discovered a unique miscalibration behavior where both correct and incorrect predictions are assigned the same level of confidence. We refer to this phenomenon as indiscriminate miscalibration. We found that traditional calibration metrics, such as Expected Calibrated Errors (ECEs), are unable to capture this behavior effectively. To address this issue, we propose new metrics to measure the severity of indiscriminate miscalibration. Additionally, we develop a novel in-context comparative inference method to alleviate miscalibrations and improve classification performance. Through extensive experiments on five datasets, we demonstrate that our proposed method can achieve more accurate and calibrated predictions compared to regular zero-shot and few-shot prompting. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_02210 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Calibrate to Discriminate: Improve In-Context Learning with Label-Free Comparative Inference Cheng, Wei Wang, Tianlu Ji, Yanmin Yang, Fan Tan, Keren Zheng, Yiyu Computation and Language Machine Learning While in-context learning with large language models (LLMs) has shown impressive performance, we have discovered a unique miscalibration behavior where both correct and incorrect predictions are assigned the same level of confidence. We refer to this phenomenon as indiscriminate miscalibration. We found that traditional calibration metrics, such as Expected Calibrated Errors (ECEs), are unable to capture this behavior effectively. To address this issue, we propose new metrics to measure the severity of indiscriminate miscalibration. Additionally, we develop a novel in-context comparative inference method to alleviate miscalibrations and improve classification performance. Through extensive experiments on five datasets, we demonstrate that our proposed method can achieve more accurate and calibrated predictions compared to regular zero-shot and few-shot prompting. |
| title | Calibrate to Discriminate: Improve In-Context Learning with Label-Free Comparative Inference |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2410.02210 |