ETF: An Entity Tracing Framework for Hallucination Detection in Code Summaries
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914024840495104 |
|---|---|
| author | Maharaj, Kishan Munigala, Vitobha Tamilselvam, Srikanth G. Kumar, Prince Sen, Sayandeep Kodeswaran, Palani Mishra, Abhijit Bhattacharyya, Pushpak |
| author_facet | Maharaj, Kishan Munigala, Vitobha Tamilselvam, Srikanth G. Kumar, Prince Sen, Sayandeep Kodeswaran, Palani Mishra, Abhijit Bhattacharyya, Pushpak |
| contents | Recent advancements in large language models (LLMs) have significantly enhanced their ability to understand both natural language and code, driving their use in tasks like natural language-to-code (NL2Code) and code summarisation. However, LLMs are prone to hallucination, outputs that stray from intended meanings. Detecting hallucinations in code summarisation is especially difficult due to the complex interplay between programming and natural languages. We introduce a first-of-its-kind dataset, CodeSumEval, with ~10K samples, curated specifically for hallucination detection in code summarisation. We further propose a novel Entity Tracing Framework (ETF) that a) utilises static program analysis to identify code entities from the program and b) uses LLMs to map and verify these entities and their intents within generated code summaries. Our experimental analysis demonstrates the framework's effectiveness, leading to a 73% F1 score. The proposed approach provides a method for detecting hallucinations by tracing entities from the summary to the code, allowing us to evaluate summary accuracy and localise the error within the summary. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_14748 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | ETF: An Entity Tracing Framework for Hallucination Detection in Code Summaries Maharaj, Kishan Munigala, Vitobha Tamilselvam, Srikanth G. Kumar, Prince Sen, Sayandeep Kodeswaran, Palani Mishra, Abhijit Bhattacharyya, Pushpak Software Engineering Artificial Intelligence Computation and Language Recent advancements in large language models (LLMs) have significantly enhanced their ability to understand both natural language and code, driving their use in tasks like natural language-to-code (NL2Code) and code summarisation. However, LLMs are prone to hallucination, outputs that stray from intended meanings. Detecting hallucinations in code summarisation is especially difficult due to the complex interplay between programming and natural languages. We introduce a first-of-its-kind dataset, CodeSumEval, with ~10K samples, curated specifically for hallucination detection in code summarisation. We further propose a novel Entity Tracing Framework (ETF) that a) utilises static program analysis to identify code entities from the program and b) uses LLMs to map and verify these entities and their intents within generated code summaries. Our experimental analysis demonstrates the framework's effectiveness, leading to a 73% F1 score. The proposed approach provides a method for detecting hallucinations by tracing entities from the summary to the code, allowing us to evaluate summary accuracy and localise the error within the summary. |
| title | ETF: An Entity Tracing Framework for Hallucination Detection in Code Summaries |
| topic | Software Engineering Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2410.14748 |