Measuring Determinism in Large Language Models for Software Code Review
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912251701624832 |
|---|---|
| author | Klishevich, Eugene Denisov-Blanch, Yegor Obstbaum, Simon Ciobanu, Igor Kosinski, Michal |
| author_facet | Klishevich, Eugene Denisov-Blanch, Yegor Obstbaum, Simon Ciobanu, Igor Kosinski, Michal |
| contents | Large Language Models (LLMs) promise to streamline software code reviews, but their ability to produce consistent assessments remains an open question. In this study, we tested four leading LLMs -- GPT-4o mini, GPT-4o, Claude 3.5 Sonnet, and LLaMA 3.2 90B Vision -- on 70 Java commits from both private and public repositories. By setting each model's temperature to zero, clearing context, and repeating the exact same prompts five times, we measured how consistently each model generated code-review assessments. Our results reveal that even with temperature minimized, LLM responses varied to different degrees. These findings highlight a consideration about the inherently limited consistency (test-retest reliability) of LLMs -- even when the temperature is set to zero -- and the need for caution when using LLM-generated code reviews to make real-world decisions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_20747 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Measuring Determinism in Large Language Models for Software Code Review Klishevich, Eugene Denisov-Blanch, Yegor Obstbaum, Simon Ciobanu, Igor Kosinski, Michal Software Engineering Large Language Models (LLMs) promise to streamline software code reviews, but their ability to produce consistent assessments remains an open question. In this study, we tested four leading LLMs -- GPT-4o mini, GPT-4o, Claude 3.5 Sonnet, and LLaMA 3.2 90B Vision -- on 70 Java commits from both private and public repositories. By setting each model's temperature to zero, clearing context, and repeating the exact same prompts five times, we measured how consistently each model generated code-review assessments. Our results reveal that even with temperature minimized, LLM responses varied to different degrees. These findings highlight a consideration about the inherently limited consistency (test-retest reliability) of LLMs -- even when the temperature is set to zero -- and the need for caution when using LLM-generated code reviews to make real-world decisions. |
| title | Measuring Determinism in Large Language Models for Software Code Review |
| topic | Software Engineering |
| url | https://arxiv.org/abs/2502.20747 |