InfiBench: Evaluating the Question-Answering Capabilities of Code Large Language Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866912118000844800 |
|---|---|
| author | Li, Linyi Geng, Shijie Li, Zhenwen He, Yibo Yu, Hao Hua, Ziyue Ning, Guanghan Wang, Siwei Xie, Tao Yang, Hongxia |
| author_facet | Li, Linyi Geng, Shijie Li, Zhenwen He, Yibo Yu, Hao Hua, Ziyue Ning, Guanghan Wang, Siwei Xie, Tao Yang, Hongxia |
| contents | Large Language Models for code (code LLMs) have witnessed tremendous progress in recent years. With the rapid development of code LLMs, many popular evaluation benchmarks, such as HumanEval, DS-1000, and MBPP, have emerged to measure the performance of code LLMs with a particular focus on code generation tasks. However, they are insufficient to cover the full range of expected capabilities of code LLMs, which span beyond code generation to answering diverse coding-related questions. To fill this gap, we propose InfiBench, the first large-scale freeform question-answering (QA) benchmark for code to our knowledge, comprising 234 carefully selected high-quality Stack Overflow questions that span across 15 programming languages. InfiBench uses four types of model-free automatic metrics to evaluate response correctness where domain experts carefully concretize the criterion for each question. We conduct a systematic evaluation for over 100 latest code LLMs on InfiBench, leading to a series of novel and insightful findings. Our detailed analyses showcase potential directions for further advancement of code LLMs. InfiBench is fully open source at https://infi-coder.github.io/infibench and continuously expanding to foster more scientific and systematic practices for code LLM evaluation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_07940 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | InfiBench: Evaluating the Question-Answering Capabilities of Code Large Language Models Li, Linyi Geng, Shijie Li, Zhenwen He, Yibo Yu, Hao Hua, Ziyue Ning, Guanghan Wang, Siwei Xie, Tao Yang, Hongxia Software Engineering Machine Learning Large Language Models for code (code LLMs) have witnessed tremendous progress in recent years. With the rapid development of code LLMs, many popular evaluation benchmarks, such as HumanEval, DS-1000, and MBPP, have emerged to measure the performance of code LLMs with a particular focus on code generation tasks. However, they are insufficient to cover the full range of expected capabilities of code LLMs, which span beyond code generation to answering diverse coding-related questions. To fill this gap, we propose InfiBench, the first large-scale freeform question-answering (QA) benchmark for code to our knowledge, comprising 234 carefully selected high-quality Stack Overflow questions that span across 15 programming languages. InfiBench uses four types of model-free automatic metrics to evaluate response correctness where domain experts carefully concretize the criterion for each question. We conduct a systematic evaluation for over 100 latest code LLMs on InfiBench, leading to a series of novel and insightful findings. Our detailed analyses showcase potential directions for further advancement of code LLMs. InfiBench is fully open source at https://infi-coder.github.io/infibench and continuously expanding to foster more scientific and systematic practices for code LLM evaluation. |
| title | InfiBench: Evaluating the Question-Answering Capabilities of Code Large Language Models |
| topic | Software Engineering Machine Learning |
| url | https://arxiv.org/abs/2404.07940 |