Correctness isnt Efficiency: Runtime Memory Divergence in LLM-Generated Code
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866918316805718016 |
|---|---|
| author | Rajput, Prateek Song, Yewei Bonkoungou, Abdoul Aziz Olatunji, Iyiola E. Kabore, Abdoul Kader Klein, Jacques Bissyandé, Tegawendé F. |
| author_facet | Rajput, Prateek Song, Yewei Bonkoungou, Abdoul Aziz Olatunji, Iyiola E. Kabore, Abdoul Kader Klein, Jacques Bissyandé, Tegawendé F. |
| contents | Large language models (LLMs) can generate programs that pass unit tests, but passing tests does not guarantee reliable runtime behavior. We find that different correct solutions to the same task can show very different memory and performance patterns, which can lead to hidden operational risks. We present a framework to measure execution-time memory stability across multiple correct generations. At the solution level, we introduce Dynamic Mean Pairwise Distance (DMPD), which uses Dynamic Time Warping to compare the shapes of memory-usage traces after converting them into Monotonic Peak Profiles (MPPs) to reduce transient noise. Aggregating DMPD across tasks yields a model-level Model Instability Score (MIS). Experiments on BigOBench and CodeContests show substantial runtime divergence among correct solutions. Instability often increases with higher sampling temperature even when pass@1 improves. We also observe correlations between our stability measures and software engineering indicators such as cognitive and cyclomatic complexity, suggesting links between operational behavior and maintainability. Our results support stability-aware selection among passing candidates in CI/CD to reduce operational risk without sacrificing correctness. Artifacts are available. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_01215 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Correctness isnt Efficiency: Runtime Memory Divergence in LLM-Generated Code Rajput, Prateek Song, Yewei Bonkoungou, Abdoul Aziz Olatunji, Iyiola E. Kabore, Abdoul Kader Klein, Jacques Bissyandé, Tegawendé F. Software Engineering Artificial Intelligence Large language models (LLMs) can generate programs that pass unit tests, but passing tests does not guarantee reliable runtime behavior. We find that different correct solutions to the same task can show very different memory and performance patterns, which can lead to hidden operational risks. We present a framework to measure execution-time memory stability across multiple correct generations. At the solution level, we introduce Dynamic Mean Pairwise Distance (DMPD), which uses Dynamic Time Warping to compare the shapes of memory-usage traces after converting them into Monotonic Peak Profiles (MPPs) to reduce transient noise. Aggregating DMPD across tasks yields a model-level Model Instability Score (MIS). Experiments on BigOBench and CodeContests show substantial runtime divergence among correct solutions. Instability often increases with higher sampling temperature even when pass@1 improves. We also observe correlations between our stability measures and software engineering indicators such as cognitive and cyclomatic complexity, suggesting links between operational behavior and maintainability. Our results support stability-aware selection among passing candidates in CI/CD to reduce operational risk without sacrificing correctness. Artifacts are available. |
| title | Correctness isnt Efficiency: Runtime Memory Divergence in LLM-Generated Code |
| topic | Software Engineering Artificial Intelligence |
| url | https://arxiv.org/abs/2601.01215 |