Uncertainty Quantification for LLM-based Code Generation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914559561826304 |
|---|---|
| author | Xu, Senrong Tan, Yuhao Zhou, Yanke Wu, Guangyuan Li, Zenan Yao, Yuan Chen, Taolue Xu, Feng Ma, Xiaoxing |
| author_facet | Xu, Senrong Tan, Yuhao Zhou, Yanke Wu, Guangyuan Li, Zenan Yao, Yuan Chen, Taolue Xu, Feng Ma, Xiaoxing |
| contents | Prediction sets provide a theoretically grounded framework for quantifying uncertainty in machine learning models. Adapting them to structured generation tasks, in particular, large language model (LLM) based code generation, remains a challenging problem. An existing attempt proposes PAC prediction sets but is limited by its strong monotonicity assumption on risk and single-label classification framework, which severely limits the space of candidate programs and cannot accommodate the multiple valid outputs inherent to code generation. To address these limitations, we propose an approach RisCoSet that leverages multiple hypothesis testing to construct risk-controlling predictions for LLM-based code generation. Given a trained code generation model, we produce a prediction set represented by a partial program, which is guaranteed to contain a correct solution with high confidence. Extensive experiments on three LLMs demonstrate the effectiveness of the proposed method. For instance, compared with the state-of-the-art, our method can significantly reduce the code removal by up to 24.5%, at the same level of risk. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_12201 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Uncertainty Quantification for LLM-based Code Generation Xu, Senrong Tan, Yuhao Zhou, Yanke Wu, Guangyuan Li, Zenan Yao, Yuan Chen, Taolue Xu, Feng Ma, Xiaoxing Software Engineering Artificial Intelligence Prediction sets provide a theoretically grounded framework for quantifying uncertainty in machine learning models. Adapting them to structured generation tasks, in particular, large language model (LLM) based code generation, remains a challenging problem. An existing attempt proposes PAC prediction sets but is limited by its strong monotonicity assumption on risk and single-label classification framework, which severely limits the space of candidate programs and cannot accommodate the multiple valid outputs inherent to code generation. To address these limitations, we propose an approach RisCoSet that leverages multiple hypothesis testing to construct risk-controlling predictions for LLM-based code generation. Given a trained code generation model, we produce a prediction set represented by a partial program, which is guaranteed to contain a correct solution with high confidence. Extensive experiments on three LLMs demonstrate the effectiveness of the proposed method. For instance, compared with the state-of-the-art, our method can significantly reduce the code removal by up to 24.5%, at the same level of risk. |
| title | Uncertainty Quantification for LLM-based Code Generation |
| topic | Software Engineering Artificial Intelligence |
| url | https://arxiv.org/abs/2605.12201 |