Copertina

Copilot Evaluation Harness: Evaluating LLM-Guided Software Programming

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Agarwal, Anisha, Chan, Aaron, Chandel, Shubham, Jang, Jinu, Miller, Shaun, Moghaddam, Roshanak Zilouchian, Mohylevskyy, Yevhen, Sundaresan, Neel, Tufano, Michele
Natura:	Preprint
Pubblicazione:	2024
Soggetti:	Software Engineering Artificial Intelligence
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

AutoDev: Automated AI-Driven Development
di: Tufano, Michele, et al.
Pubblicazione: (2024)

RefactorBench: Evaluating Stateful Reasoning in Language Agents Through Code
di: Gautam, Dhruv, et al.
Pubblicazione: (2025)

SetupBench: Assessing Software Engineering Agents' Ability to Bootstrap Development Environments
di: Arora, Avi, et al.
Pubblicazione: (2025)

PerfBench: Can Agents Resolve Real-World Performance Bugs?
di: Garg, Spandan, et al.
Pubblicazione: (2025)

RAPGen: An Approach for Fixing Code Inefficiencies in Zero-Shot
di: Garg, Spandan, et al.
Pubblicazione: (2023)

Closing the Gap: A User Study on the Real-world Usefulness of AI-powered Vulnerability Detection & Repair in the IDE
di: Steenhoek, Benjamin, et al.
Pubblicazione: (2024)

FaultLine: Automated Proof-of-Vulnerability Generation Using LLM Agents
di: Nitin, Vikram, et al.
Pubblicazione: (2025)

The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason
di: Liang, Shanchao, et al.
Pubblicazione: (2025)

When Developer Aid Becomes Security Debt: A Systematic Analysis of Insecure Behaviors in LLM Coding Agents
di: Kozak, Matous, et al.
Pubblicazione: (2025)

Reinforcement Learning from Automatic Feedback for High-Quality Unit Test Generation
di: Steenhoek, Benjamin, et al.
Pubblicazione: (2023)

Reinforcement Learning from Automatic Feedback for High-Quality Unit Test Generation
di: Steenhoek, Benjamin, et al.
Pubblicazione: (2024)

Evaluating Agent-based Program Repair at Google
di: Rondon, Pat, et al.
Pubblicazione: (2025)

NU-Class Net: A Novel Approach for Video Quality Enhancement
di: Moghaddam, Parham Zilouchian, et al.
Pubblicazione: (2024)

Evaluating Step-by-step Reasoning Traces: A Survey
di: Lee, Jinu, et al.
Pubblicazione: (2025)

One prompt is not enough: Instruction Sensitivity Undermines Embedding Model Evaluation
di: Kostiuk, Yevhen, et al.
Pubblicazione: (2026)

The Impact of Generative AI on Collaborative Open-Source Software Development: Evidence from GitHub Copilot
di: Song, Fangchen, et al.
Pubblicazione: (2024)

Transforming Software Development: Evaluating the Efficiency and Challenges of GitHub Copilot in Real-World Projects
di: Pandey, Ruchika, et al.
Pubblicazione: (2024)

A User-centered Security Evaluation of Copilot
di: Asare, Owura, et al.
Pubblicazione: (2023)

Agentic Bug Reproduction for Effective Automated Program Repair at Google
di: Cheng, Runxiang, et al.
Pubblicazione: (2025)

ED-Copilot: Reduce Emergency Department Wait Time with Language Model Diagnostic Assistance
di: Sun, Liwen, et al.
Pubblicazione: (2024)

Toward a Sustainable Software Architecture Community: Evaluating ICSA's Environmental Impact
di: Moghaddam, Mahyar T., et al.
Pubblicazione: (2026)

Programming with AI: Evaluating ChatGPT, Gemini, AlphaCode, and GitHub Copilot for Programmers
di: Siam, Md Kamrul, et al.
Pubblicazione: (2024)

Chapter Percorsi familiari e preminenza a Nola alla fine del Medioevo. Il caso degli Albertini di Cimitile
di: Tufano, Luigi
Pubblicazione: (2022)

Chapter Potere feudale ed élite locale nel Mezzogiorno alla fine del Medioevo. Note sulla contea orsiniana di Nola
di: Tufano, Luigi
Pubblicazione: (2022)

Towards a Human-in-the-Loop Framework for Reliable Patch Evaluation Using an LLM-as-a-Judge
di: Shi, Sherry, et al.
Pubblicazione: (2025)

Artifact: I have no idea how to make it safer: Studying Security and Privacy Mindsets of Browser Extension Developers
di: Agarwal, Shubham
Pubblicazione: (2025)

Breakloose suppression in minimal friction models
di: Agarwal, Shubham
Pubblicazione: (2026)

Evaluating the tipping point of a complex system: The case of disruptive technology
di: Christine M. Edwards, et al.
Pubblicazione: (2024)

On Software Ageing Indicators in OpenStack
di: Yazvinskyi, Yevhen, et al.
Pubblicazione: (2024)

Comparative Evaluation of Canal Transport and Centralization Between ProTaper Next and XP‐endo Shaper Systems Using CBCT Analysis: An In Vitro Study
di: Hamed Karkehabadi, et al.
Pubblicazione: (2025)

Copilot Arena: A Platform for Code LLM Evaluation in the Wild
di: Chi, Wayne, et al.
Pubblicazione: (2025)

Towards Multilingual LLM Evaluation for Baltic and Nordic languages: A study on Lithuanian History
di: Kostiuk, Yevhen, et al.
Pubblicazione: (2025)

The Veln(ia)s is in the Details: Evaluating LLM Judgment on Latvian and Lithuanian Short Answer Matching
di: Kostiuk, Yevhen, et al.
Pubblicazione: (2025)

Poster Presentation: Cultural Omnivorousness in the Domains of Music, Film and Literature: Evidence for a Partial Overlap [2025]
di: Voronin, Yevhen
Pubblicazione: (2025)

Replication files [Detailing Social Influence in Predicting Cinema Attendance: a Vignette Approach (2025) [Poetics]] [Stata]
di: Voronin, Yevhen
Pubblicazione: (2025)

THE DEVELOPMENT OF UKRAINE'S CRIMINAL LAW POLICY UNDER MARTIAL LAW AND EXPERT REFLECTIONS ON IT
di: Pysmenskyy, Yevhen
Pubblicazione: (2025)

Framework for asset-liability management with fixed-term securities
di: Havrylenko, Yevhen
Pubblicazione: (2025)

The main pedagogical aspects of the formation of professional competence of students of physical culture
di: Yevhen Prystupa
Pubblicazione: (2022)

Experimental Study of Action Different Kinetic Energy on the Colon
di: Yevhen Kvasnevskyi
Pubblicazione: (2022)

THEORY AND LEGAL REGULATION OF INFORMATION SUPPORT OF ADMINISTRATIVE PROCEDURES IN UKRAINE
di: Yevhen Leheza
Pubblicazione: (2021)