Simple and Effective Baselines for Code Summarisation Evaluation
Fuente:
arXiv
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910968496259072 |
|---|---|
| author | Robinson, Jade Kummerfeld, Jonathan K. |
| author_facet | Robinson, Jade Kummerfeld, Jonathan K. |
| contents | Code documentation is useful, but writing it is time-consuming. Different techniques for generating code summaries have emerged, but comparing them is difficult because human evaluation is expensive and automatic metrics are unreliable. In this paper, we introduce a simple new baseline in which we ask an LLM to give an overall score to a summary. Unlike n-gram and embedding-based baselines, our approach is able to consider the code when giving a score. This allows us to also make a variant that does not consider the reference summary at all, which could be used for other tasks, e.g., to evaluate the quality of documentation in code bases. We find that our method is as good or better than prior metrics, though we recommend using it in conjunction with embedding-based methods to avoid the risk of LLM-specific bias. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_19392 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Simple and Effective Baselines for Code Summarisation Evaluation Robinson, Jade Kummerfeld, Jonathan K. Computation and Language Artificial Intelligence Software Engineering 68T50 I.2.7 Code documentation is useful, but writing it is time-consuming. Different techniques for generating code summaries have emerged, but comparing them is difficult because human evaluation is expensive and automatic metrics are unreliable. In this paper, we introduce a simple new baseline in which we ask an LLM to give an overall score to a summary. Unlike n-gram and embedding-based baselines, our approach is able to consider the code when giving a score. This allows us to also make a variant that does not consider the reference summary at all, which could be used for other tasks, e.g., to evaluate the quality of documentation in code bases. We find that our method is as good or better than prior metrics, though we recommend using it in conjunction with embedding-based methods to avoid the risk of LLM-specific bias. |
| title | Simple and Effective Baselines for Code Summarisation Evaluation |
| topic | Computation and Language Artificial Intelligence Software Engineering 68T50 I.2.7 |
| url | https://arxiv.org/abs/2505.19392 |