Impact of LLM-based Review Comment Generation in Practice: A Mixed Open-/Closed-source User Study
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866929587615694848 |
|---|---|
| author | Olewicki, Doriane Da Silva, Leuson Mujahid, Suhaib Amini, Arezou Mah, Benjamin Castelluccio, Marco Habchi, Sarra Khomh, Foutse Adams, Bram |
| author_facet | Olewicki, Doriane Da Silva, Leuson Mujahid, Suhaib Amini, Arezou Mah, Benjamin Castelluccio, Marco Habchi, Sarra Khomh, Foutse Adams, Bram |
| contents | We conduct a large-scale empirical user study in a live setup to evaluate the acceptance of LLM-generated comments and their impact on the review process. This user study was performed in two organizations, Mozilla (which has its codebase available as open source) and Ubisoft (fully closed-source). Inside their usual review environment, participants were given access to RevMate, an LLM-based assistive tool suggesting generated review comments using an off-the-shelf LLM with Retrieval Augmented Generation to provide extra code and review context, combined with LLM-as-a-Judge, to auto-evaluate the generated comments and discard irrelevant cases. Based on more than 587 patch reviews provided by RevMate, we observed that 8.1% and 7.2%, respectively, of LLM-generated comments were accepted by reviewers in each organization, while 14.6% and 20.5% other comments were still marked as valuable as review or development tips. Refactoring-related comments are more likely to be accepted than Functional comments (18.2% and 18.6% compared to 4.8% and 5.2%). The extra time spent by reviewers to inspect generated comments or edit accepted ones (36/119), yielding an overall median of 43s per patch, is reasonable. The accepted generated comments are as likely to yield future revisions of the revised patch as human-written comments (74% vs 73% at chunk-level). |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_07091 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Impact of LLM-based Review Comment Generation in Practice: A Mixed Open-/Closed-source User Study Olewicki, Doriane Da Silva, Leuson Mujahid, Suhaib Amini, Arezou Mah, Benjamin Castelluccio, Marco Habchi, Sarra Khomh, Foutse Adams, Bram Software Engineering We conduct a large-scale empirical user study in a live setup to evaluate the acceptance of LLM-generated comments and their impact on the review process. This user study was performed in two organizations, Mozilla (which has its codebase available as open source) and Ubisoft (fully closed-source). Inside their usual review environment, participants were given access to RevMate, an LLM-based assistive tool suggesting generated review comments using an off-the-shelf LLM with Retrieval Augmented Generation to provide extra code and review context, combined with LLM-as-a-Judge, to auto-evaluate the generated comments and discard irrelevant cases. Based on more than 587 patch reviews provided by RevMate, we observed that 8.1% and 7.2%, respectively, of LLM-generated comments were accepted by reviewers in each organization, while 14.6% and 20.5% other comments were still marked as valuable as review or development tips. Refactoring-related comments are more likely to be accepted than Functional comments (18.2% and 18.6% compared to 4.8% and 5.2%). The extra time spent by reviewers to inspect generated comments or edit accepted ones (36/119), yielding an overall median of 43s per patch, is reasonable. The accepted generated comments are as likely to yield future revisions of the revised patch as human-written comments (74% vs 73% at chunk-level). |
| title | Impact of LLM-based Review Comment Generation in Practice: A Mixed Open-/Closed-source User Study |
| topic | Software Engineering |
| url | https://arxiv.org/abs/2411.07091 |