Tricky$^2$: Towards a Benchmark for Evaluating Human and LLM Error Interactions
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911401639936000 |
|---|---|
| author | Granger, Cole Khati, Dipin Rodriguez-Cardenas, Daniel Poshyvanyk, Denys |
| author_facet | Granger, Cole Khati, Dipin Rodriguez-Cardenas, Daniel Poshyvanyk, Denys |
| contents | Large language models (LLMs) are increasingly integrated into software development workflows, yet they often introduce subtle logic or data-misuse errors that differ from human bugs. To study how these two error types interact, we construct Tricky$^2$, a hybrid dataset that augments the existing TrickyBugs corpus of human-written defects with errors injected by both GPT-5 and OpenAI-oss-20b across C++, Python, and Java programs. Our approach uses a taxonomy-guided prompting framework to generate machine-originated bugs while preserving original human defects and program structure. The resulting corpus spans human-only, LLM-only, and human+LLM splits, enabling analysis of mixed-origin error behavior, multi-bug repair robustness, and reliability in hybrid human-machine code. This paper outlines the dataset construction pipeline and illustrates its use through small-scale baseline evaluations of classification, localization, and repair tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_18949 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Tricky$^2$: Towards a Benchmark for Evaluating Human and LLM Error Interactions Granger, Cole Khati, Dipin Rodriguez-Cardenas, Daniel Poshyvanyk, Denys Software Engineering Artificial Intelligence Large language models (LLMs) are increasingly integrated into software development workflows, yet they often introduce subtle logic or data-misuse errors that differ from human bugs. To study how these two error types interact, we construct Tricky$^2$, a hybrid dataset that augments the existing TrickyBugs corpus of human-written defects with errors injected by both GPT-5 and OpenAI-oss-20b across C++, Python, and Java programs. Our approach uses a taxonomy-guided prompting framework to generate machine-originated bugs while preserving original human defects and program structure. The resulting corpus spans human-only, LLM-only, and human+LLM splits, enabling analysis of mixed-origin error behavior, multi-bug repair robustness, and reliability in hybrid human-machine code. This paper outlines the dataset construction pipeline and illustrates its use through small-scale baseline evaluations of classification, localization, and repair tasks. |
| title | Tricky$^2$: Towards a Benchmark for Evaluating Human and LLM Error Interactions |
| topic | Software Engineering Artificial Intelligence |
| url | https://arxiv.org/abs/2601.18949 |