The RealHumanEval: Evaluating Large Language Models' Abilities to Support Programmers
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913546272505856 |
|---|---|
| author | Mozannar, Hussein Chen, Valerie Alsobay, Mohammed Das, Subhro Zhao, Sebastian Wei, Dennis Nagireddy, Manish Sattigeri, Prasanna Talwalkar, Ameet Sontag, David |
| author_facet | Mozannar, Hussein Chen, Valerie Alsobay, Mohammed Das, Subhro Zhao, Sebastian Wei, Dennis Nagireddy, Manish Sattigeri, Prasanna Talwalkar, Ameet Sontag, David |
| contents | Evaluation of large language models for code has primarily relied on static benchmarks, including HumanEval (Chen et al., 2021), or more recently using human preferences of LLM responses. As LLMs are increasingly used as programmer assistants, we study whether gains on existing benchmarks or more preferred LLM responses translate to programmer productivity when coding with LLMs, including time spent coding. We introduce RealHumanEval, a web interface to measure the ability of LLMs to assist programmers, through either autocomplete or chat support. We conducted a user study (N=243) using RealHumanEval in which users interacted with seven LLMs of varying base model performance. Despite static benchmarks not incorporating humans-in-the-loop, we find that improvements in benchmark performance lead to increased programmer productivity; however gaps in benchmark versus human performance are not proportional -- a trend that holds across both forms of LLM support. In contrast, we find that programmer preferences do not correlate with their actual performance, motivating the need for better proxy signals. We open-source RealHumanEval to enable human-centric evaluation of new models and the study data to facilitate efforts to improve code models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_02806 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | The RealHumanEval: Evaluating Large Language Models' Abilities to Support Programmers Mozannar, Hussein Chen, Valerie Alsobay, Mohammed Das, Subhro Zhao, Sebastian Wei, Dennis Nagireddy, Manish Sattigeri, Prasanna Talwalkar, Ameet Sontag, David Software Engineering Artificial Intelligence Human-Computer Interaction Evaluation of large language models for code has primarily relied on static benchmarks, including HumanEval (Chen et al., 2021), or more recently using human preferences of LLM responses. As LLMs are increasingly used as programmer assistants, we study whether gains on existing benchmarks or more preferred LLM responses translate to programmer productivity when coding with LLMs, including time spent coding. We introduce RealHumanEval, a web interface to measure the ability of LLMs to assist programmers, through either autocomplete or chat support. We conducted a user study (N=243) using RealHumanEval in which users interacted with seven LLMs of varying base model performance. Despite static benchmarks not incorporating humans-in-the-loop, we find that improvements in benchmark performance lead to increased programmer productivity; however gaps in benchmark versus human performance are not proportional -- a trend that holds across both forms of LLM support. In contrast, we find that programmer preferences do not correlate with their actual performance, motivating the need for better proxy signals. We open-source RealHumanEval to enable human-centric evaluation of new models and the study data to facilitate efforts to improve code models. |
| title | The RealHumanEval: Evaluating Large Language Models' Abilities to Support Programmers |
| topic | Software Engineering Artificial Intelligence Human-Computer Interaction |
| url | https://arxiv.org/abs/2404.02806 |