Document Retrieval Augmented Fine-Tuning (DRAFT) for safety-critical software assessments
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917475637002240 |
|---|---|
| author | Bolton, Regan Sheikhfathollahi, Mohammadreza Parkinson, Simon Vulovic, Vanessa Bamford, Gary Basher, Dan Parkinson, Howard |
| author_facet | Bolton, Regan Sheikhfathollahi, Mohammadreza Parkinson, Simon Vulovic, Vanessa Bamford, Gary Basher, Dan Parkinson, Howard |
| contents | Safety critical software assessment requires robust assessment against complex regulatory frameworks, a process traditionally limited by manual evaluation. This paper presents Document Retrieval-Augmented Fine-Tuning (DRAFT), a novel approach that enhances the capabilities of a large language model (LLM) for safety-critical compliance assessment. DRAFT builds upon existing Retrieval-Augmented Generation (RAG) techniques by introducing a novel fine-tuning framework that accommodates our dual-retrieval architecture, which simultaneously accesses both software documentation and applicable reference standards. To fine-tune DRAFT, we develop a semi-automated dataset generation methodology that incorporates variable numbers of relevant documents with meaningful distractors, closely mirroring real-world assessment scenarios. Experiments with GPT-4o-mini demonstrate a 7% improvement in correctness over the baseline model, with qualitative improvements in evidence handling, response structure, and domain-specific reasoning. DRAFT represents a practical approach to improving compliance assessment systems while maintaining the transparency and evidence-based reasoning essential in regulatory domains. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_01307 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Document Retrieval Augmented Fine-Tuning (DRAFT) for safety-critical software assessments Bolton, Regan Sheikhfathollahi, Mohammadreza Parkinson, Simon Vulovic, Vanessa Bamford, Gary Basher, Dan Parkinson, Howard Software Engineering Artificial Intelligence Safety critical software assessment requires robust assessment against complex regulatory frameworks, a process traditionally limited by manual evaluation. This paper presents Document Retrieval-Augmented Fine-Tuning (DRAFT), a novel approach that enhances the capabilities of a large language model (LLM) for safety-critical compliance assessment. DRAFT builds upon existing Retrieval-Augmented Generation (RAG) techniques by introducing a novel fine-tuning framework that accommodates our dual-retrieval architecture, which simultaneously accesses both software documentation and applicable reference standards. To fine-tune DRAFT, we develop a semi-automated dataset generation methodology that incorporates variable numbers of relevant documents with meaningful distractors, closely mirroring real-world assessment scenarios. Experiments with GPT-4o-mini demonstrate a 7% improvement in correctness over the baseline model, with qualitative improvements in evidence handling, response structure, and domain-specific reasoning. DRAFT represents a practical approach to improving compliance assessment systems while maintaining the transparency and evidence-based reasoning essential in regulatory domains. |
| title | Document Retrieval Augmented Fine-Tuning (DRAFT) for safety-critical software assessments |
| topic | Software Engineering Artificial Intelligence |
| url | https://arxiv.org/abs/2505.01307 |