Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2402.05939 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929238115876864 |
|---|---|
| author | Li, Yufei Chen, Simin Guo, Yanghong Yang, Wei Dong, Yue Liu, Cong |
| author_facet | Li, Yufei Chen, Simin Guo, Yanghong Yang, Wei Dong, Yue Liu, Cong |
| contents | Large Language Models (LLMs) have been widely employed in programming language analysis to enhance human productivity. Yet, their reliability can be compromised by various code distribution shifts, leading to inconsistent outputs. While probabilistic methods are known to mitigate such impact through uncertainty calibration and estimation, their efficacy in the language domain remains underexplored compared to their application in image-based tasks. In this work, we first introduce a large-scale benchmark dataset, incorporating three realistic patterns of code distribution shifts at varying intensities. Then we thoroughly investigate state-of-the-art probabilistic methods applied to CodeLlama using these shifted code snippets. We observe that these methods generally improve the uncertainty awareness of CodeLlama, with increased calibration quality and higher uncertainty estimation~(UE) precision. However, our study further reveals varied performance dynamics across different criteria (e.g., calibration error vs misclassification detection) and trade-off between efficacy and efficiency, highlighting necessary methodological selection tailored to specific contexts. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_05939 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Uncertainty Awareness of Large Language Models Under Code Distribution Shifts: A Benchmark Study Li, Yufei Chen, Simin Guo, Yanghong Yang, Wei Dong, Yue Liu, Cong Software Engineering Computation and Language Machine Learning Large Language Models (LLMs) have been widely employed in programming language analysis to enhance human productivity. Yet, their reliability can be compromised by various code distribution shifts, leading to inconsistent outputs. While probabilistic methods are known to mitigate such impact through uncertainty calibration and estimation, their efficacy in the language domain remains underexplored compared to their application in image-based tasks. In this work, we first introduce a large-scale benchmark dataset, incorporating three realistic patterns of code distribution shifts at varying intensities. Then we thoroughly investigate state-of-the-art probabilistic methods applied to CodeLlama using these shifted code snippets. We observe that these methods generally improve the uncertainty awareness of CodeLlama, with increased calibration quality and higher uncertainty estimation~(UE) precision. However, our study further reveals varied performance dynamics across different criteria (e.g., calibration error vs misclassification detection) and trade-off between efficacy and efficiency, highlighting necessary methodological selection tailored to specific contexts. |
| title | Uncertainty Awareness of Large Language Models Under Code Distribution Shifts: A Benchmark Study |
| topic | Software Engineering Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2402.05939 |