Saved in:
| Main Author: | |
|---|---|
| Format: | Recurso digital |
| Language: | |
| Published: |
Zenodo
2025
|
| Online Access: | https://doi.org/10.5281/zenodo.17209143 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866901502778408960 |
|---|---|
| author | anon |
| author_facet | anon |
| contents | <h1>Replication Package for Historian</h1> <p>This repository provides the replication package "Historian".</p> <h2>Overview</h2> <p>Historian is a tool to tackle this by formulating patch assessment as code clone detection and semantic similarity. It introduces a novel paradigm that formulates APCA as a multi-reference code clone detection problem against a historical knowledge base.</p> <p><a title="Historian Overview" href="https://anonymous.4open.science/api/repo/Historian-Artifact/file/rebutal/method.png">https://anonymous.4open.science/api/repo/Historian-Artifact/file/rebutal/method.png</a></p> <ul> <li> <p><strong>Evidence-Based Decision Making</strong>: Unlike black-box classifiers, Historian provides transparent verdicts by pointing to specific historical patches that match, enabling expert verification.</p> </li> <li> <p><strong>Self-Improving Architecture</strong>: The reference set grows over time, making Historian progressively more powerful without expensive model retraining.</p> </li> <li> <p><strong>Principled Uncertainty Handling</strong>: Unknown labels indicate insufficient historical evidence, directing human experts to cases requiring thorough evaluation.</p> </li> <li> <p><strong>Performance</strong>: Historian demonstrates superior performance compared to existing SOTA methods (Acc/F1 W. Avg. 88.4/0.91 in Table 6 compared to 84.0/0.88 of SOTA in Table 7).</p> </li> </ul> <p><a title="Historian Performance" href="https://anonymous.4open.science/api/repo/Historian-Artifact/file/rebutal/main-results.png">https://anonymous.4open.science/api/repo/Historian-Artifact/file/rebutal/main-results.png</a></p> <h2>Purpose and Scope</h2> <p>In the error-prone research environment of APR, our empirical studies show that many APR-generated patches are repetitive and exhibit human-understandable similarities (Type-1, Type-2 clones), making relabeling from scratch inefficient and unnecessary. The tool addresses the specific challenge of reducing redundant validation effort while maintaining high accuracy through evidence-based decision making.</p> <h2>Requirements</h2> <ul> <li>Python 3.10</li> <li>Java 1.8</li> <li>OS: Tested on Ubuntu Linux</li> </ul> <p>Install Python dependencies:</p> <pre><code>pip install -r requirements.txt </code></pre> <p>Initialize submodules:</p> <pre><code>git submodule update --init --recursive </code></pre> <h2>Project Structure</h2> <pre><code>├── benchmarks/ # bug benchmarks used for evaluation │ ├── bears/ # Bears benchmark │ ├── benchmarks.json # benchmarks configuration file │ ├── bugsjar/ # Bugs.jar benchmark │ ├── defects4j/ # Defects4J benchmark │ ├── ID2commit-bugsjar/ # commit ID mappings for Bugs.jar │ ├── introclassjava/ # IntroClassJava benchmark │ └── quixbugs/ # QuixBugs benchmark ├── build.py # script to generate the files in tmp/results (RQ3 and RQ4) ├── classify.py # script to classify LLM responses ├── datasets/ # historically validated APR patches │ ├── aprenfl/ # APR-ENF-L dataset │ ├── datasets.json # datasets configuration file with links to datasets │ ├── defectrepairing/ # DefectRepairing dataset │ ├── dl4pc2/ # DL4PC2 dataset │ ├── drr/ # DRR dataset │ └── wangicse/ # Wang ICSE dataset ├── __pycache__/ # Python cache files │ └── build.cpython-310.pyc # compiled Python bytecode ├── rebutal/ # includes materials mentioned in rebuttal response ├── requirements.txt # Python dependencies ├── results.json # refeneces to LLM responses ├── results.py # script to aggregate (majority voting) results and generate plots ├── rq1_plots.py # script to generate plots for research question 1 ├── rq1.py # script to generate results for research question 1 ├── rq3_zeroshot_plots.py # script to generate zero-shot classification performance plots ├── tmp/ # results, logs, and intermediate files │ ├── checkouts/ # includes all generated documents │ ├── data/ # includes preprocessed data in each step │ ├── logs/ # execution logs │ ├── methods/ # extracted methods storage │ ├── patches/ # cleaned patches storage │ ├── plots/ # generated figures │ └── results/ # raw LLM responses and classification results ├── tools/ # tools for patch analysis │ ├── matching/ # AST-based code clone detection tool │ ├── ollama/ # Ollama configurations │ ├── SourcererCC/ # SourcererCC code clone detection tool (text-based) │ └── tools.json # list of tools and links └── utils/ # helper utilities for preprocessing and analysis ├── benchmark.py # API access to benchmark metadata ├── config.py # configuration (paths, keys) ├── dataset.py # interfaces for dataset access ├── __pycache__/ # Python cache files ├── tool.py # scripts to wrap tools └── utils.py # utils </code></pre> <h3>Key Subdirectories in tmp/:</h3> <ul> <li><code>tmp/data/metadata/</code>: includes datasets, benchmarks and other metadata used in experiments</li> <li><code>tmp/data/metadata/ollama/</code>: includes prompts and models used in experiments</li> <li><code>tmp/results/classification/</code>: classified LLM outputs for research questions 3 and 4 (keywords EXP2 and EXP3 respectively)</li> <li><code>tmp/results/rq1/</code>: labels for research question 1</li> <li><code>tmp/results/expert/</code>: expert labels for research question 2 (EXP2-*.pkl files)</li> </ul> <h2>RQ1</h2> <p>To generate results of the first research question:</p> <pre><code>python rq1.py python rq1_plots.py </code></pre> <p>This will save the classified responses in <code>tmp/results/classified/</code> and generate plots in <code>tmp/plots/</code>.</p> <h2>RQ2, RQ3 and RQ4</h2> <p><code>tmp/results/</code> includes raw LLM responses in pickle files. To classify the LLM responses, aggregate votes and generate summary plots for 2nd, 3rd and 4th research questions:</p> <pre><code>python results.py </code></pre> <p>Results will be saved in <code>tmp/plots/</code>.</p> <h2>RQ3</h2> <ul> <li>Zero-Shot Classification Performance:</li> </ul> <pre><code>python rq3_zeroshot_plots.py </code></pre> <p>Results are saved in tmp/plots/rq3</p> <h2>Regenerate Raw LLM Responses</h2> <p>Raw LLM responses are stored in <code>tmp/results/</code> as pickle files.</p> <p>If you wish to generate LLM responses from scratch, instructions are provided in the following:</p> <p>To reproduce the LLM responses if they do not already exist in <code>tmp/results/</code>, ensure that Ollama (https://ollama.com/) is installed, the server is running and the desired models are pulled using:</p> <pre><code>ollama pull <model_name> </code></pre> <p>The list of required models is specified in <code>tmp/data/metadata/ollama/models.json</code>.</p> <p>Then run the following:</p> <pre><code>python build.py </code></pre> <h1>Clean Patches and Methods</h1> <p>Cleaned patches and extracted methods are stored in <code>tmp/patches</code> and <code>tmp/methods</code> respectively. If you wish to regenerate them out of <code>datasets</code>, consider the following.</p> <p>Initialize submodules:</p> <pre><code>git submodule update --init --recursive </code></pre> <p>Add Defects4J (https://github.com/rjust/defects4j) in <code>benchmarks/defects4j</code> to the path and initialize it. For detailed instructions, please visit Defects4J (https://github.com/rjust/defects4j):</p> <p>Add IDs to Bugs.jar:</p> <pre><code>cp -r benchmarks/ID2commit-bugsjar benchmarks/bugsjar/ID2commit </code></pre> <p>Then run:</p> <pre><code>python build.py </code></pre> <p>Please note that this script will skip cleaning and extraction phases if the corresponding pickle files exist in <code>tmp/data</code>. These pickle files are generated (by <code>build.py</code>) after extraction and include metadata (e.g., location and ID of patches, methods, bugs, etc.)</p> |
| format | Recurso digital |
| id | zenodo_https___doi_org_10_5281_zenodo_17209143 |
| institution | Zenodo |
| language | |
| publishDate | 2025 |
| publisher | Zenodo |
| record_format | zenodo |
| spellingShingle | Historian: Leveraging Community Validated Patches to Determine Patch Correctness anon <h1>Replication Package for Historian</h1> <p>This repository provides the replication package "Historian".</p> <h2>Overview</h2> <p>Historian is a tool to tackle this by formulating patch assessment as code clone detection and semantic similarity. It introduces a novel paradigm that formulates APCA as a multi-reference code clone detection problem against a historical knowledge base.</p> <p><a title="Historian Overview" href="https://anonymous.4open.science/api/repo/Historian-Artifact/file/rebutal/method.png">https://anonymous.4open.science/api/repo/Historian-Artifact/file/rebutal/method.png</a></p> <ul> <li> <p><strong>Evidence-Based Decision Making</strong>: Unlike black-box classifiers, Historian provides transparent verdicts by pointing to specific historical patches that match, enabling expert verification.</p> </li> <li> <p><strong>Self-Improving Architecture</strong>: The reference set grows over time, making Historian progressively more powerful without expensive model retraining.</p> </li> <li> <p><strong>Principled Uncertainty Handling</strong>: Unknown labels indicate insufficient historical evidence, directing human experts to cases requiring thorough evaluation.</p> </li> <li> <p><strong>Performance</strong>: Historian demonstrates superior performance compared to existing SOTA methods (Acc/F1 W. Avg. 88.4/0.91 in Table 6 compared to 84.0/0.88 of SOTA in Table 7).</p> </li> </ul> <p><a title="Historian Performance" href="https://anonymous.4open.science/api/repo/Historian-Artifact/file/rebutal/main-results.png">https://anonymous.4open.science/api/repo/Historian-Artifact/file/rebutal/main-results.png</a></p> <h2>Purpose and Scope</h2> <p>In the error-prone research environment of APR, our empirical studies show that many APR-generated patches are repetitive and exhibit human-understandable similarities (Type-1, Type-2 clones), making relabeling from scratch inefficient and unnecessary. The tool addresses the specific challenge of reducing redundant validation effort while maintaining high accuracy through evidence-based decision making.</p> <h2>Requirements</h2> <ul> <li>Python 3.10</li> <li>Java 1.8</li> <li>OS: Tested on Ubuntu Linux</li> </ul> <p>Install Python dependencies:</p> <pre><code>pip install -r requirements.txt </code></pre> <p>Initialize submodules:</p> <pre><code>git submodule update --init --recursive </code></pre> <h2>Project Structure</h2> <pre><code>├── benchmarks/ # bug benchmarks used for evaluation │ ├── bears/ # Bears benchmark │ ├── benchmarks.json # benchmarks configuration file │ ├── bugsjar/ # Bugs.jar benchmark │ ├── defects4j/ # Defects4J benchmark │ ├── ID2commit-bugsjar/ # commit ID mappings for Bugs.jar │ ├── introclassjava/ # IntroClassJava benchmark │ └── quixbugs/ # QuixBugs benchmark ├── build.py # script to generate the files in tmp/results (RQ3 and RQ4) ├── classify.py # script to classify LLM responses ├── datasets/ # historically validated APR patches │ ├── aprenfl/ # APR-ENF-L dataset │ ├── datasets.json # datasets configuration file with links to datasets │ ├── defectrepairing/ # DefectRepairing dataset │ ├── dl4pc2/ # DL4PC2 dataset │ ├── drr/ # DRR dataset │ └── wangicse/ # Wang ICSE dataset ├── __pycache__/ # Python cache files │ └── build.cpython-310.pyc # compiled Python bytecode ├── rebutal/ # includes materials mentioned in rebuttal response ├── requirements.txt # Python dependencies ├── results.json # refeneces to LLM responses ├── results.py # script to aggregate (majority voting) results and generate plots ├── rq1_plots.py # script to generate plots for research question 1 ├── rq1.py # script to generate results for research question 1 ├── rq3_zeroshot_plots.py # script to generate zero-shot classification performance plots ├── tmp/ # results, logs, and intermediate files │ ├── checkouts/ # includes all generated documents │ ├── data/ # includes preprocessed data in each step │ ├── logs/ # execution logs │ ├── methods/ # extracted methods storage │ ├── patches/ # cleaned patches storage │ ├── plots/ # generated figures │ └── results/ # raw LLM responses and classification results ├── tools/ # tools for patch analysis │ ├── matching/ # AST-based code clone detection tool │ ├── ollama/ # Ollama configurations │ ├── SourcererCC/ # SourcererCC code clone detection tool (text-based) │ └── tools.json # list of tools and links └── utils/ # helper utilities for preprocessing and analysis ├── benchmark.py # API access to benchmark metadata ├── config.py # configuration (paths, keys) ├── dataset.py # interfaces for dataset access ├── __pycache__/ # Python cache files ├── tool.py # scripts to wrap tools └── utils.py # utils </code></pre> <h3>Key Subdirectories in tmp/:</h3> <ul> <li><code>tmp/data/metadata/</code>: includes datasets, benchmarks and other metadata used in experiments</li> <li><code>tmp/data/metadata/ollama/</code>: includes prompts and models used in experiments</li> <li><code>tmp/results/classification/</code>: classified LLM outputs for research questions 3 and 4 (keywords EXP2 and EXP3 respectively)</li> <li><code>tmp/results/rq1/</code>: labels for research question 1</li> <li><code>tmp/results/expert/</code>: expert labels for research question 2 (EXP2-*.pkl files)</li> </ul> <h2>RQ1</h2> <p>To generate results of the first research question:</p> <pre><code>python rq1.py python rq1_plots.py </code></pre> <p>This will save the classified responses in <code>tmp/results/classified/</code> and generate plots in <code>tmp/plots/</code>.</p> <h2>RQ2, RQ3 and RQ4</h2> <p><code>tmp/results/</code> includes raw LLM responses in pickle files. To classify the LLM responses, aggregate votes and generate summary plots for 2nd, 3rd and 4th research questions:</p> <pre><code>python results.py </code></pre> <p>Results will be saved in <code>tmp/plots/</code>.</p> <h2>RQ3</h2> <ul> <li>Zero-Shot Classification Performance:</li> </ul> <pre><code>python rq3_zeroshot_plots.py </code></pre> <p>Results are saved in tmp/plots/rq3</p> <h2>Regenerate Raw LLM Responses</h2> <p>Raw LLM responses are stored in <code>tmp/results/</code> as pickle files.</p> <p>If you wish to generate LLM responses from scratch, instructions are provided in the following:</p> <p>To reproduce the LLM responses if they do not already exist in <code>tmp/results/</code>, ensure that Ollama (https://ollama.com/) is installed, the server is running and the desired models are pulled using:</p> <pre><code>ollama pull <model_name> </code></pre> <p>The list of required models is specified in <code>tmp/data/metadata/ollama/models.json</code>.</p> <p>Then run the following:</p> <pre><code>python build.py </code></pre> <h1>Clean Patches and Methods</h1> <p>Cleaned patches and extracted methods are stored in <code>tmp/patches</code> and <code>tmp/methods</code> respectively. If you wish to regenerate them out of <code>datasets</code>, consider the following.</p> <p>Initialize submodules:</p> <pre><code>git submodule update --init --recursive </code></pre> <p>Add Defects4J (https://github.com/rjust/defects4j) in <code>benchmarks/defects4j</code> to the path and initialize it. For detailed instructions, please visit Defects4J (https://github.com/rjust/defects4j):</p> <p>Add IDs to Bugs.jar:</p> <pre><code>cp -r benchmarks/ID2commit-bugsjar benchmarks/bugsjar/ID2commit </code></pre> <p>Then run:</p> <pre><code>python build.py </code></pre> <p>Please note that this script will skip cleaning and extraction phases if the corresponding pickle files exist in <code>tmp/data</code>. These pickle files are generated (by <code>build.py</code>) after extraction and include metadata (e.g., location and ID of patches, methods, bugs, etc.)</p> |
| title | Historian: Leveraging Community Validated Patches to Determine Patch Correctness |
| url | https://doi.org/10.5281/zenodo.17209143 |