Improving Code Search with Hard Negative Sampling Based on Fine-tuning

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Dong, Hande, Lin, Jiayi, Wang, Yanlin, Leng, Yichong, Chen, Jiawei, Xie, Yutao
Format:	Preprint
Published:	2023
Subjects:	Software Engineering
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

_version_	1866913582890876928
author	Dong, Hande Lin, Jiayi Wang, Yanlin Leng, Yichong Chen, Jiawei Xie, Yutao
author_facet	Dong, Hande Lin, Jiayi Wang, Yanlin Leng, Yichong Chen, Jiawei Xie, Yutao
contents	Pre-trained code models have emerged as the state-of-the-art paradigm for code search tasks. The paradigm involves pre-training the model on search-irrelevant tasks such as masked language modeling, followed by the fine-tuning stage, which focuses on the search-relevant task. The typical fine-tuning method is to employ a dual-encoder architecture to encode semantic embeddings of query and code separately, and then calculate their similarity based on the embeddings. However, the typical dual-encoder architecture falls short in modeling token-level interactions between query and code, which limits the capabilities of model. To address this limitation, we introduce a cross-encoder architecture for code search that jointly encodes the concatenation of query and code. We further introduce a Retriever-Ranker (RR) framework that cascades the dual-encoder and cross-encoder to promote the efficiency of evaluation and online serving. Moreover, we present a ranking-based hard negative sampling (PS) method to improve the ability of cross-encoder to distinguish hard negative codes, which further enhances the cascaded RR framework. Experiments on four datasets using three code models demonstrate the superiority of our proposed method. We have made the code available at https://github.com/DongHande/R2PS.
format	Preprint
id	arxiv_https___arxiv_org_abs_2305_04508
institution	arXiv
publishDate	2023
record_format	arxiv
spellingShingle	Improving Code Search with Hard Negative Sampling Based on Fine-tuning Dong, Hande Lin, Jiayi Wang, Yanlin Leng, Yichong Chen, Jiawei Xie, Yutao Software Engineering Pre-trained code models have emerged as the state-of-the-art paradigm for code search tasks. The paradigm involves pre-training the model on search-irrelevant tasks such as masked language modeling, followed by the fine-tuning stage, which focuses on the search-relevant task. The typical fine-tuning method is to employ a dual-encoder architecture to encode semantic embeddings of query and code separately, and then calculate their similarity based on the embeddings. However, the typical dual-encoder architecture falls short in modeling token-level interactions between query and code, which limits the capabilities of model. To address this limitation, we introduce a cross-encoder architecture for code search that jointly encodes the concatenation of query and code. We further introduce a Retriever-Ranker (RR) framework that cascades the dual-encoder and cross-encoder to promote the efficiency of evaluation and online serving. Moreover, we present a ranking-based hard negative sampling (PS) method to improve the ability of cross-encoder to distinguish hard negative codes, which further enhances the cascaded RR framework. Experiments on four datasets using three code models demonstrate the superiority of our proposed method. We have made the code available at https://github.com/DongHande/R2PS.
title	Improving Code Search with Hard Negative Sampling Based on Fine-tuning
topic	Software Engineering
url	https://arxiv.org/abs/2305.04508