Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2406.11589 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918321680547840 |
|---|---|
| author | Gong, Jing Wu, Yanghui Liang, Linxi Wang, Yanlin Chen, Jiachi Liu, Mingwei Zheng, Zibin |
| author_facet | Gong, Jing Wu, Yanghui Liang, Linxi Wang, Yanlin Chen, Jiachi Liu, Mingwei Zheng, Zibin |
| contents | Semantic code search, retrieving code that matches a given natural language query, is an important task to improve productivity in software engineering. Existing code search datasets face limitations: they rely on human annotators who assess code primarily through semantic understanding rather than functional verification, leading to potential inaccuracies and scalability issues. Additionally, current evaluation metrics often overlook the multi-choice nature of code search. This paper introduces CoSQA+, pairing high-quality queries from CoSQA with multiple suitable codes. We develop an automated pipeline featuring multiple model-based candidate selections and the novel test-driven agent annotation system. Among a single Large Language Model (LLM) annotator and Python expert annotators (without test-based verification), agents leverage test-based verification and achieve the highest accuracy of 93.9%. Through extensive experiments, CoSQA+ has demonstrated superior quality over CoSQA. Models trained on CoSQA+ exhibit improved performance. We publicly release both CoSQA+_all, which contains 412,080 agent-annotated pairs, and CoSQA+_verified, which contains 1,000 human-verified pairs, at https://github.com/DeepSoftwareAnalytics/CoSQA_Plus. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_11589 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | CoSQA+: Pioneering the Multi-Choice Code Search Benchmark with Test-Driven Agents Gong, Jing Wu, Yanghui Liang, Linxi Wang, Yanlin Chen, Jiachi Liu, Mingwei Zheng, Zibin Software Engineering Artificial Intelligence Information Retrieval I.2.7; D.2.3 Semantic code search, retrieving code that matches a given natural language query, is an important task to improve productivity in software engineering. Existing code search datasets face limitations: they rely on human annotators who assess code primarily through semantic understanding rather than functional verification, leading to potential inaccuracies and scalability issues. Additionally, current evaluation metrics often overlook the multi-choice nature of code search. This paper introduces CoSQA+, pairing high-quality queries from CoSQA with multiple suitable codes. We develop an automated pipeline featuring multiple model-based candidate selections and the novel test-driven agent annotation system. Among a single Large Language Model (LLM) annotator and Python expert annotators (without test-based verification), agents leverage test-based verification and achieve the highest accuracy of 93.9%. Through extensive experiments, CoSQA+ has demonstrated superior quality over CoSQA. Models trained on CoSQA+ exhibit improved performance. We publicly release both CoSQA+_all, which contains 412,080 agent-annotated pairs, and CoSQA+_verified, which contains 1,000 human-verified pairs, at https://github.com/DeepSoftwareAnalytics/CoSQA_Plus. |
| title | CoSQA+: Pioneering the Multi-Choice Code Search Benchmark with Test-Driven Agents |
| topic | Software Engineering Artificial Intelligence Information Retrieval I.2.7; D.2.3 |
| url | https://arxiv.org/abs/2406.11589 |