Leveraging LLMs for Legacy Code Modernization: Challenges and Opportunities for LLM-Generated Documentation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929600828801024 |
|---|---|
| author | Diggs, Colin Doyle, Michael Madan, Amit Scott, Siggy Escamilla, Emily Zimmer, Jacob Nekoo, Naveed Ursino, Paul Bartholf, Michael Robin, Zachary Patel, Anand Glasz, Chris Macke, William Kirk, Paul Phillips, Jasper Sridharan, Arun Wendt, Doug Rosen, Scott Naik, Nitin Brunelle, Justin F. Thaker, Samruddhi |
| author_facet | Diggs, Colin Doyle, Michael Madan, Amit Scott, Siggy Escamilla, Emily Zimmer, Jacob Nekoo, Naveed Ursino, Paul Bartholf, Michael Robin, Zachary Patel, Anand Glasz, Chris Macke, William Kirk, Paul Phillips, Jasper Sridharan, Arun Wendt, Doug Rosen, Scott Naik, Nitin Brunelle, Justin F. Thaker, Samruddhi |
| contents | Legacy software systems, written in outdated languages like MUMPS and mainframe assembly, pose challenges in efficiency, maintenance, staffing, and security. While LLMs offer promise for modernizing these systems, their ability to understand legacy languages is largely unknown. This paper investigates the utilization of LLMs to generate documentation for legacy code using two datasets: an electronic health records (EHR) system in MUMPS and open-source applications in IBM mainframe Assembly Language Code (ALC). We propose a prompting strategy for generating line-wise code comments and a rubric to evaluate their completeness, readability, usefulness, and hallucination. Our study assesses the correlation between human evaluations and automated metrics, such as code complexity and reference-based metrics. We find that LLM-generated comments for MUMPS and ALC are generally hallucination-free, complete, readable, and useful compared to ground-truth comments, though ALC poses challenges. However, no automated metrics strongly correlate with comment quality to predict or measure LLM performance. Our findings highlight the limitations of current automated measures and the need for better evaluation metrics for LLM-generated documentation in legacy systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_14971 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Leveraging LLMs for Legacy Code Modernization: Challenges and Opportunities for LLM-Generated Documentation Diggs, Colin Doyle, Michael Madan, Amit Scott, Siggy Escamilla, Emily Zimmer, Jacob Nekoo, Naveed Ursino, Paul Bartholf, Michael Robin, Zachary Patel, Anand Glasz, Chris Macke, William Kirk, Paul Phillips, Jasper Sridharan, Arun Wendt, Doug Rosen, Scott Naik, Nitin Brunelle, Justin F. Thaker, Samruddhi Machine Learning Software Engineering Legacy software systems, written in outdated languages like MUMPS and mainframe assembly, pose challenges in efficiency, maintenance, staffing, and security. While LLMs offer promise for modernizing these systems, their ability to understand legacy languages is largely unknown. This paper investigates the utilization of LLMs to generate documentation for legacy code using two datasets: an electronic health records (EHR) system in MUMPS and open-source applications in IBM mainframe Assembly Language Code (ALC). We propose a prompting strategy for generating line-wise code comments and a rubric to evaluate their completeness, readability, usefulness, and hallucination. Our study assesses the correlation between human evaluations and automated metrics, such as code complexity and reference-based metrics. We find that LLM-generated comments for MUMPS and ALC are generally hallucination-free, complete, readable, and useful compared to ground-truth comments, though ALC poses challenges. However, no automated metrics strongly correlate with comment quality to predict or measure LLM performance. Our findings highlight the limitations of current automated measures and the need for better evaluation metrics for LLM-generated documentation in legacy systems. |
| title | Leveraging LLMs for Legacy Code Modernization: Challenges and Opportunities for LLM-Generated Documentation |
| topic | Machine Learning Software Engineering |
| url | https://arxiv.org/abs/2411.14971 |