An Empirical Analysis of Static Analysis Methods for Detection and Mitigation of Code Library Hallucinations
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913020649668608 |
|---|---|
| author | Miranda-Pena, Clarissa Reeson, Andrew Paris, Cécile Poon, Josiah Kummerfeld, Jonathan K. |
| author_facet | Miranda-Pena, Clarissa Reeson, Andrew Paris, Cécile Poon, Josiah Kummerfeld, Jonathan K. |
| contents | Despite extensive research, Large Language Models continue to hallucinate when generating code, particularly when using libraries. On NL-to-code benchmarks that require library use, we find that LLMs generate code that uses non-existent library features in 8.1-40% of responses. One intuitive approach for detection and mitigation of hallucinations is static analysis. In this paper, we analyse the potential of static analysis tools, both in terms of what they can solve and what they cannot. We find that static analysis tools can detect 16-70% of all errors, and 14-85% of library hallucinations, with performance varying by LLM and dataset. Through manual analysis, we identify cases a static method could not plausibly catch, which gives an upper bound on their potential from 48.5% to 77%. Overall, we show that static analysis methods are cheap method for addressing some forms of hallucination, and we quantify how far short of solving the problem they will always be. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_07755 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | An Empirical Analysis of Static Analysis Methods for Detection and Mitigation of Code Library Hallucinations Miranda-Pena, Clarissa Reeson, Andrew Paris, Cécile Poon, Josiah Kummerfeld, Jonathan K. Computation and Language Software Engineering Despite extensive research, Large Language Models continue to hallucinate when generating code, particularly when using libraries. On NL-to-code benchmarks that require library use, we find that LLMs generate code that uses non-existent library features in 8.1-40% of responses. One intuitive approach for detection and mitigation of hallucinations is static analysis. In this paper, we analyse the potential of static analysis tools, both in terms of what they can solve and what they cannot. We find that static analysis tools can detect 16-70% of all errors, and 14-85% of library hallucinations, with performance varying by LLM and dataset. Through manual analysis, we identify cases a static method could not plausibly catch, which gives an upper bound on their potential from 48.5% to 77%. Overall, we show that static analysis methods are cheap method for addressing some forms of hallucination, and we quantify how far short of solving the problem they will always be. |
| title | An Empirical Analysis of Static Analysis Methods for Detection and Mitigation of Code Library Hallucinations |
| topic | Computation and Language Software Engineering |
| url | https://arxiv.org/abs/2604.07755 |