Salvato in:
| Autore principale: | |
|---|---|
| Natura: | Recurso digital |
| Lingua: | |
| Pubblicazione: |
Zenodo
2025
|
| Accesso online: | https://doi.org/10.5281/zenodo.15030382 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
Sommario:
- <h1>UntrustVul Replication Package</h1> <p>Improving the Usability of Vulnerability Detection Models by Reducing Untrustworthy Alerts</p> <h2>Dataset</h2> <p>We publish 12 trustworthiness datasets of 4 models (LineVul, SVulD, IVDetect, and ReVeal) on 3 vulnerability datasets (BigVul, MegaVul, SARD). The datasets are located at <code>data/trust</code>. Each file is in csv format, with the following columns:</p> <ul> <li><code>Source Index</code>: the index of the source code</li> <li><code>Code</code>: the source code</li> <li><code>Confidence</code>: the model confidence of the vulnerability prediction</li> <li><code>Expl Line</code>: the line numbers that contribute the most to the prediction</li> <li><code>Expl Score</code>: the corresponding importance scores</li> <li><code>Flaw Line Index</code>: the ground-truth indices of flaw lines</li> <li><code>Flaw Lines</code>: the ground-truth flaw lines</li> <li><code>Filter Flaw Lines</code>: the flaw lines excluding redundant lines</li> <li><code>gt</code>: the ground-truth trustworthiness label, <code>0</code> for untrustworthy and <code>1</code> for trustworthy</li> </ul> <p>Datasets used for training line-level assessment models are in <code>data/line</code>.</p> <h2>1. Data Preprocessing</h2> <p>Please extract each source code (each row in the dataset) into a file in the format of <code><source_index>.c</code> and put them in a directory named <code><detector_name></code> for each csv file in <code>data/trust</code>. We use Joern as a tool to parse the source code into a program dependency graph (PDG). You can use both the <a href="https://github.com/octopus-platform/joern">old version</a> of Joern or the <a href="https://github.com/joernio/joern">new one</a></p> <h3>Old version</h3> <p>For source codes in a directory, run <code>joern-parse outputDirectory srcDirectory</code> to get the csv files.</p> <h3>New version</h3> <p>For each source file, run <code>joern --script data_preprocess/export_graph.sc --param file=<src_file> --param output=<json_output_file></code>.</p> <p>The old version outputs 2 csv files for each source file storing nodes and edges, while the new version outputs a json file.</p> <h2>2. Training models for line-level assessment</h2> <p>To train the model, run the following commands:</p> <pre><code>python line_checking/trainer.py \ --data_dir data/line \ --output_dir <path_to_model_out_dir> \ --model_type roberta \ --tokenizer_name <insert_your_tokenizer_name> \ --model_name_or_path <insert_your_model_name> \ --epochs 10 \ --block_size 128 \ --train_batch_size 16 \ --eval_batch_size 16 \ --learning_rate 2e-5 \ --max_grad_norm 1.0 \ --evaluate_during_training \ --seed 123456 </code></pre> <p><code>tokenizer_name</code> and <code>model_name_or_path</code> can be <code>microsoft/unixcoder-base</code>, <code>microsoft/graphcodebert-base</code>, <code>microsoft/codebert-base</code>, etc.</p> <h2>3. Trustworthiness Evaluation</h2> <p>To evaluate the trustworthiness of vulnerability predictions, run the following commands:</p> <pre><code>python main.py \ --input_data <path_to_trust_data> \ --model_file_path <path_to_saved_file_of_model1> <path_to_saved_file_of_model2> ... <path_to_saved_file_of_modeln> \ --model_name_or_path <model_name1> <model_name2> ... <model_namen> \ --detector <name_of_detector> \ --parsed_type <csv_or_json> \ --parsed_path <directory_containing_parsed_results> </code></pre> <p>Trust datasets are stored in <code>data\trust</code>. Param <code>--model_file_path</code> points to the line-level assessment models from Step 2. Detector can be <code>linevul</code>, <code>svuld</code>, <code>ivdetect</code>, or <code>reveal</code>. Param <code>--parsed_path</code> points to the directory storing parsed results from Step 1.</p> <h2>4. Improving Models</h2> <p>Please refer to the replicate package of <a href="https://dl.acm.org/doi/10.1145/3597503.3639170">CausalVul</a>.</p> <p><em>Because of the randomness, the evaluation results may be slightly different.</em></p>