Verbatim Data Transcription Failures in LLM Code Generation: A State-Tracking Stress Test
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915712786759680 |
|---|---|
| author | Haque, Mohd Ariful Gupta, Kishor Datta Rahman, Mohammad Ashiqur George, Roy |
| author_facet | Haque, Mohd Ariful Gupta, Kishor Datta Rahman, Mohammad Ashiqur George, Roy |
| contents | Many real-world software tasks require exact transcription of provided data into code, such as cryptographic constants, protocol test vectors, allowlists, and calibration tables. These tasks are operationally sensitive because small omissions or alterations can remain silent while producing syntactically valid programs. This paper introduces a deliberately minimal transcription-to-code benchmark to isolate this reliability concern in LLM-based code generation. Given a list of high-precision decimal constants, a model must generate Python code that embeds the constants verbatim and performs a simple aggregate computation. We describe the prompting variants, evaluation protocol based on exact-string inclusion, and analysis framework used to characterize state-tracking and long-horizon generation failures. The benchmark is intended as a compact stress test that complements existing code-generation evaluations by focusing on data integrity rather than algorithmic reasoning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_03640 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Verbatim Data Transcription Failures in LLM Code Generation: A State-Tracking Stress Test Haque, Mohd Ariful Gupta, Kishor Datta Rahman, Mohammad Ashiqur George, Roy Software Engineering Cryptography and Security Many real-world software tasks require exact transcription of provided data into code, such as cryptographic constants, protocol test vectors, allowlists, and calibration tables. These tasks are operationally sensitive because small omissions or alterations can remain silent while producing syntactically valid programs. This paper introduces a deliberately minimal transcription-to-code benchmark to isolate this reliability concern in LLM-based code generation. Given a list of high-precision decimal constants, a model must generate Python code that embeds the constants verbatim and performs a simple aggregate computation. We describe the prompting variants, evaluation protocol based on exact-string inclusion, and analysis framework used to characterize state-tracking and long-horizon generation failures. The benchmark is intended as a compact stress test that complements existing code-generation evaluations by focusing on data integrity rather than algorithmic reasoning. |
| title | Verbatim Data Transcription Failures in LLM Code Generation: A State-Tracking Stress Test |
| topic | Software Engineering Cryptography and Security |
| url | https://arxiv.org/abs/2601.03640 |