RepoQA: Evaluating Long Context Code Understanding
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917689199427584 |
|---|---|
| author | Liu, Jiawei Tian, Jia Le Daita, Vijay Wei, Yuxiang Ding, Yifeng Wang, Yuhan Katherine Yang, Jun Zhang, Lingming |
| author_facet | Liu, Jiawei Tian, Jia Le Daita, Vijay Wei, Yuxiang Ding, Yifeng Wang, Yuhan Katherine Yang, Jun Zhang, Lingming |
| contents | Recent advances have been improving the context windows of Large Language Models (LLMs). To quantify the real long-context capabilities of LLMs, evaluators such as the popular Needle in a Haystack have been developed to test LLMs over a large chunk of raw texts. While effective, current evaluations overlook the insight of how LLMs work with long-context code, i.e., repositories. To this end, we initiate the RepoQA benchmark to evaluate LLMs on long-context code understanding. Traditional needle testers ask LLMs to directly retrieve the answer from the context without necessary deep understanding. In RepoQA, we built our initial task, namely Searching Needle Function (SNF), which exercises LLMs to search functions given their natural-language description, i.e., LLMs cannot find the desired function if they cannot understand the description and code. RepoQA is multilingual and comprehensive: it includes 500 code search tasks gathered from 50 popular repositories across 5 modern programming languages. By evaluating 26 general and code-specific LLMs on RepoQA, we show (i) there is still a small gap between the best open and proprietary models; (ii) different models are good at different languages; and (iii) models may understand code better without comments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_06025 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | RepoQA: Evaluating Long Context Code Understanding Liu, Jiawei Tian, Jia Le Daita, Vijay Wei, Yuxiang Ding, Yifeng Wang, Yuhan Katherine Yang, Jun Zhang, Lingming Software Engineering Computation and Language Machine Learning Recent advances have been improving the context windows of Large Language Models (LLMs). To quantify the real long-context capabilities of LLMs, evaluators such as the popular Needle in a Haystack have been developed to test LLMs over a large chunk of raw texts. While effective, current evaluations overlook the insight of how LLMs work with long-context code, i.e., repositories. To this end, we initiate the RepoQA benchmark to evaluate LLMs on long-context code understanding. Traditional needle testers ask LLMs to directly retrieve the answer from the context without necessary deep understanding. In RepoQA, we built our initial task, namely Searching Needle Function (SNF), which exercises LLMs to search functions given their natural-language description, i.e., LLMs cannot find the desired function if they cannot understand the description and code. RepoQA is multilingual and comprehensive: it includes 500 code search tasks gathered from 50 popular repositories across 5 modern programming languages. By evaluating 26 general and code-specific LLMs on RepoQA, we show (i) there is still a small gap between the best open and proprietary models; (ii) different models are good at different languages; and (iii) models may understand code better without comments. |
| title | RepoQA: Evaluating Long Context Code Understanding |
| topic | Software Engineering Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2406.06025 |