Copilot Arena: A Platform for Code LLM Evaluation in the Wild
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915150270824448 |
|---|---|
| author | Chi, Wayne Chen, Valerie Angelopoulos, Anastasios Nikolas Chiang, Wei-Lin Mittal, Aditya Jain, Naman Zhang, Tianjun Stoica, Ion Donahue, Chris Talwalkar, Ameet |
| author_facet | Chi, Wayne Chen, Valerie Angelopoulos, Anastasios Nikolas Chiang, Wei-Lin Mittal, Aditya Jain, Naman Zhang, Tianjun Stoica, Ion Donahue, Chris Talwalkar, Ameet |
| contents | Evaluating in-the-wild coding capabilities of large language models (LLMs) is a challenging endeavor with no clear solution. We introduce Copilot Arena, a platform to collect user preferences for code generation through native integration into a developer's working environment. Copilot Arena comprises a novel interface for comparing pairs of model outputs, a sampling strategy optimized to reduce latency, and a prompting scheme to enable code completion functionality. Copilot Arena has served over 4.5 million suggestions from 10 models and collected over 11k pairwise judgements. Our results highlight the importance of model evaluations in integrated settings. We find that model rankings from Copilot Arena differ from those of existing evaluations, which we attribute to the more realistic distribution of data and tasks contained in Copilot Arena. We also identify novel insights into human preferences on code such as an observed consistency in user preference across programming languages yet significant variation in preference due to task category. We open-source Copilot Arena and release data to enable human-centric evaluations and improve understanding of coding assistants. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_09328 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Copilot Arena: A Platform for Code LLM Evaluation in the Wild Chi, Wayne Chen, Valerie Angelopoulos, Anastasios Nikolas Chiang, Wei-Lin Mittal, Aditya Jain, Naman Zhang, Tianjun Stoica, Ion Donahue, Chris Talwalkar, Ameet Software Engineering Evaluating in-the-wild coding capabilities of large language models (LLMs) is a challenging endeavor with no clear solution. We introduce Copilot Arena, a platform to collect user preferences for code generation through native integration into a developer's working environment. Copilot Arena comprises a novel interface for comparing pairs of model outputs, a sampling strategy optimized to reduce latency, and a prompting scheme to enable code completion functionality. Copilot Arena has served over 4.5 million suggestions from 10 models and collected over 11k pairwise judgements. Our results highlight the importance of model evaluations in integrated settings. We find that model rankings from Copilot Arena differ from those of existing evaluations, which we attribute to the more realistic distribution of data and tasks contained in Copilot Arena. We also identify novel insights into human preferences on code such as an observed consistency in user preference across programming languages yet significant variation in preference due to task category. We open-source Copilot Arena and release data to enable human-centric evaluations and improve understanding of coding assistants. |
| title | Copilot Arena: A Platform for Code LLM Evaluation in the Wild |
| topic | Software Engineering |
| url | https://arxiv.org/abs/2502.09328 |