Evaluating the Ability of Large Language Models to Generate Verifiable Specifications in VeriFast
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913633881030656 |
|---|---|
| author | Fan, Wen Rego, Marilyn Hu, Xin Dod, Sanya Ni, Zhaorui Xie, Danning DiVincenzo, Jenna Tan, Lin |
| author_facet | Fan, Wen Rego, Marilyn Hu, Xin Dod, Sanya Ni, Zhaorui Xie, Danning DiVincenzo, Jenna Tan, Lin |
| contents | Static verification is a powerful method for enhancing software quality, but it demands significant human labor and resources. This is particularly true of static verifiers that reason about heap manipulating programs using an ownership logic. LLMs have shown promise in a number of software engineering activities, including code generation, test generation, proof generation for theorem provers, and specification generation for static verifiers. However, prior work has not explored how well LLMs can perform specification generation for specifications based in an ownership logic, such as separation logic. To address this gap, this paper explores OpenAI's GPT-4o model's effectiveness in generating specifications on C programs that are verifiable with VeriFast, a separation logic based static verifier. Our experiment employs three different types of user inputs as well as basic and Chain-of-Thought (CoT) prompting to assess GPT's capabilities. Our results indicate that the specifications generated by GPT-4o preserve functional behavior, but struggle to be verifiable. When the specifications are verifiable they contain redundancies. Future directions are discussed to improve the performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_02318 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Evaluating the Ability of Large Language Models to Generate Verifiable Specifications in VeriFast Fan, Wen Rego, Marilyn Hu, Xin Dod, Sanya Ni, Zhaorui Xie, Danning DiVincenzo, Jenna Tan, Lin Software Engineering Artificial Intelligence Logic in Computer Science Programming Languages Static verification is a powerful method for enhancing software quality, but it demands significant human labor and resources. This is particularly true of static verifiers that reason about heap manipulating programs using an ownership logic. LLMs have shown promise in a number of software engineering activities, including code generation, test generation, proof generation for theorem provers, and specification generation for static verifiers. However, prior work has not explored how well LLMs can perform specification generation for specifications based in an ownership logic, such as separation logic. To address this gap, this paper explores OpenAI's GPT-4o model's effectiveness in generating specifications on C programs that are verifiable with VeriFast, a separation logic based static verifier. Our experiment employs three different types of user inputs as well as basic and Chain-of-Thought (CoT) prompting to assess GPT's capabilities. Our results indicate that the specifications generated by GPT-4o preserve functional behavior, but struggle to be verifiable. When the specifications are verifiable they contain redundancies. Future directions are discussed to improve the performance. |
| title | Evaluating the Ability of Large Language Models to Generate Verifiable Specifications in VeriFast |
| topic | Software Engineering Artificial Intelligence Logic in Computer Science Programming Languages |
| url | https://arxiv.org/abs/2411.02318 |