Replication Package of "A multi-dimensional characterization of the Defects4J benchmark for analyzing APR evaluation bias"
Fuente:
Zenodo
Saved in:
| Main Author: | |
|---|---|
| Format: | Recurso digital |
| Published: |
Zenodo
2026
|
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866901649218338816 |
|---|---|
| author | Anonymous Anonymous |
| author_facet | Anonymous Anonymous |
| contents | <p class="font-claude-response-body break-words whitespace-normal leading-[1.7]">Replication package for "A multi-dimensional characterization of the Defects4J benchmark for analyzing APR evaluation bias". This repository contains the full pipeline for characterizing bugs from the Defects4J benchmark across six dimensions, along with the final dataset, validation results, and generated figures.</p> <p class="font-claude-response-body break-words whitespace-normal leading-[1.7]">The final dataset covers 854 bugs across 17 Java projects.</p> <p class="font-claude-response-body break-words whitespace-normal leading-[1.7]"><strong>Repository Structure</strong></p> <ul> <li><code class="bg-text-200/5 border border-0.5 border-border-300 text-danger-000 whitespace-pre-wrap rounded-[0.4rem] px-1 py-px text-[0.9rem]">scripts/</code> — Pipeline scripts for bug information extraction, characterization, validation, and analysis</li> <li><code class="bg-text-200/5 border border-0.5 border-border-300 text-danger-000 whitespace-pre-wrap rounded-[0.4rem] px-1 py-px text-[0.9rem]">data/d4j_characterization.csv</code> — Final characterization dataset (854 bugs)</li> <li><code class="bg-text-200/5 border border-0.5 border-border-300 text-danger-000 whitespace-pre-wrap rounded-[0.4rem] px-1 py-px text-[0.9rem]">data/issue_reports_html/</code> — Issue reports from issue trackers(e.g., JIRA, GitHub)</li> <li><code class="bg-text-200/5 border border-0.5 border-border-300 text-danger-000 whitespace-pre-wrap rounded-[0.4rem] px-1 py-px text-[0.9rem]">data/d4j_repo_extraction/</code> — Per-bug JSON summaries from Defects4J</li> <li><code class="bg-text-200/5 border border-0.5 border-border-300 text-danger-000 whitespace-pre-wrap rounded-[0.4rem] px-1 py-px text-[0.9rem]">data/d4j_bugreport_extraction/</code> — Per-bug JSON summaries from issue reports</li> <li><code class="bg-text-200/5 border border-0.5 border-border-300 text-danger-000 whitespace-pre-wrap rounded-[0.4rem] px-1 py-px text-[0.9rem]">data/validation_sample.csv</code> — 105 bugs selected for human validation</li> <li><code class="bg-text-200/5 border border-0.5 border-border-300 text-danger-000 whitespace-pre-wrap rounded-[0.4rem] px-1 py-px text-[0.9rem]">data/validation_annotation.csv</code> — Human-labelled corrections where the LLM's labels differ</li> <li><code class="bg-text-200/5 border border-0.5 border-border-300 text-danger-000 whitespace-pre-wrap rounded-[0.4rem] px-1 py-px text-[0.9rem]">data/validation_results.csv</code> — LLM-human agreement results</li> <li><code class="bg-text-200/5 border border-0.5 border-border-300 text-danger-000 whitespace-pre-wrap rounded-[0.4rem] px-1 py-px text-[0.9rem]">charts/</code> — Generated figures (PNG + PDF)</li> </ul> <p class="font-claude-response-body break-words whitespace-normal leading-[1.7]"><strong>Requirements</strong></p> <ul> <li>Python 3.8+, Java 11</li> <li>Defects4J repository (for extraction only)</li> <li>Azure OpenAI access (for characterization scripts)</li> </ul> <p class="font-claude-response-body break-words whitespace-normal leading-[1.7]"> See <code class="bg-text-200/5 border border-0.5 border-border-300 text-danger-000 whitespace-pre-wrap rounded-[0.4rem] px-1 py-px text-[0.9rem]">README.md</code> inside the package for full step-by-step instructions.</p> |
| format | Recurso digital |
| id | zenodo_https___doi_org_10_5281_zenodo_19975931 |
| institution | Zenodo |
| language | |
| publishDate | 2026 |
| publisher | Zenodo |
| record_format | zenodo |
| spellingShingle | Replication Package of "A multi-dimensional characterization of the Defects4J benchmark for analyzing APR evaluation bias" Anonymous Anonymous <p class="font-claude-response-body break-words whitespace-normal leading-[1.7]">Replication package for "A multi-dimensional characterization of the Defects4J benchmark for analyzing APR evaluation bias". This repository contains the full pipeline for characterizing bugs from the Defects4J benchmark across six dimensions, along with the final dataset, validation results, and generated figures.</p> <p class="font-claude-response-body break-words whitespace-normal leading-[1.7]">The final dataset covers 854 bugs across 17 Java projects.</p> <p class="font-claude-response-body break-words whitespace-normal leading-[1.7]"><strong>Repository Structure</strong></p> <ul> <li><code class="bg-text-200/5 border border-0.5 border-border-300 text-danger-000 whitespace-pre-wrap rounded-[0.4rem] px-1 py-px text-[0.9rem]">scripts/</code> — Pipeline scripts for bug information extraction, characterization, validation, and analysis</li> <li><code class="bg-text-200/5 border border-0.5 border-border-300 text-danger-000 whitespace-pre-wrap rounded-[0.4rem] px-1 py-px text-[0.9rem]">data/d4j_characterization.csv</code> — Final characterization dataset (854 bugs)</li> <li><code class="bg-text-200/5 border border-0.5 border-border-300 text-danger-000 whitespace-pre-wrap rounded-[0.4rem] px-1 py-px text-[0.9rem]">data/issue_reports_html/</code> — Issue reports from issue trackers(e.g., JIRA, GitHub)</li> <li><code class="bg-text-200/5 border border-0.5 border-border-300 text-danger-000 whitespace-pre-wrap rounded-[0.4rem] px-1 py-px text-[0.9rem]">data/d4j_repo_extraction/</code> — Per-bug JSON summaries from Defects4J</li> <li><code class="bg-text-200/5 border border-0.5 border-border-300 text-danger-000 whitespace-pre-wrap rounded-[0.4rem] px-1 py-px text-[0.9rem]">data/d4j_bugreport_extraction/</code> — Per-bug JSON summaries from issue reports</li> <li><code class="bg-text-200/5 border border-0.5 border-border-300 text-danger-000 whitespace-pre-wrap rounded-[0.4rem] px-1 py-px text-[0.9rem]">data/validation_sample.csv</code> — 105 bugs selected for human validation</li> <li><code class="bg-text-200/5 border border-0.5 border-border-300 text-danger-000 whitespace-pre-wrap rounded-[0.4rem] px-1 py-px text-[0.9rem]">data/validation_annotation.csv</code> — Human-labelled corrections where the LLM's labels differ</li> <li><code class="bg-text-200/5 border border-0.5 border-border-300 text-danger-000 whitespace-pre-wrap rounded-[0.4rem] px-1 py-px text-[0.9rem]">data/validation_results.csv</code> — LLM-human agreement results</li> <li><code class="bg-text-200/5 border border-0.5 border-border-300 text-danger-000 whitespace-pre-wrap rounded-[0.4rem] px-1 py-px text-[0.9rem]">charts/</code> — Generated figures (PNG + PDF)</li> </ul> <p class="font-claude-response-body break-words whitespace-normal leading-[1.7]"><strong>Requirements</strong></p> <ul> <li>Python 3.8+, Java 11</li> <li>Defects4J repository (for extraction only)</li> <li>Azure OpenAI access (for characterization scripts)</li> </ul> <p class="font-claude-response-body break-words whitespace-normal leading-[1.7]"> See <code class="bg-text-200/5 border border-0.5 border-border-300 text-danger-000 whitespace-pre-wrap rounded-[0.4rem] px-1 py-px text-[0.9rem]">README.md</code> inside the package for full step-by-step instructions.</p> |
| title | Replication Package of "A multi-dimensional characterization of the Defects4J benchmark for analyzing APR evaluation bias" |
| url | https://doi.org/10.5281/zenodo.19975931 |