Towards Translating Real-World Code with LLMs: A Study of Translating to Rust
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912332911738880 |
|---|---|
| author | Eniser, Hasan Ferit Zhang, Hanliang David, Cristina Wang, Meng Christakis, Maria Paulsen, Brandon Dodds, Joey Kroening, Daniel |
| author_facet | Eniser, Hasan Ferit Zhang, Hanliang David, Cristina Wang, Meng Christakis, Maria Paulsen, Brandon Dodds, Joey Kroening, Daniel |
| contents | Large language models (LLMs) show promise in code translation - the task of translating code written in one programming language to another language - due to their ability to write code in most programming languages. However, LLM's effectiveness on translating real-world code remains largely unstudied. In this work, we perform the first substantial study on LLM-based translation to Rust by assessing the ability of five state-of-the-art LLMs, GPT4, Claude 3, Claude 2.1, Gemini Pro, and Mixtral. We conduct our study on code extracted from real-world open source projects. To enable our study, we develop FLOURINE, an end-to-end code translation tool that uses differential fuzzing to check if a Rust translation is I/O equivalent to the original source program, eliminating the need for pre-existing test cases. As part of our investigation, we assess both the LLM's ability to produce an initially successful translation, as well as their capacity to fix a previously generated buggy one. If the original and the translated programs are not I/O equivalent, we apply a set of automated feedback strategies, including feedback to the LLM with counterexamples. Our results show that the most successful LLM can translate 47% of our benchmarks, and also provides insights into next steps for improvements. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_11514 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Towards Translating Real-World Code with LLMs: A Study of Translating to Rust Eniser, Hasan Ferit Zhang, Hanliang David, Cristina Wang, Meng Christakis, Maria Paulsen, Brandon Dodds, Joey Kroening, Daniel Software Engineering Large language models (LLMs) show promise in code translation - the task of translating code written in one programming language to another language - due to their ability to write code in most programming languages. However, LLM's effectiveness on translating real-world code remains largely unstudied. In this work, we perform the first substantial study on LLM-based translation to Rust by assessing the ability of five state-of-the-art LLMs, GPT4, Claude 3, Claude 2.1, Gemini Pro, and Mixtral. We conduct our study on code extracted from real-world open source projects. To enable our study, we develop FLOURINE, an end-to-end code translation tool that uses differential fuzzing to check if a Rust translation is I/O equivalent to the original source program, eliminating the need for pre-existing test cases. As part of our investigation, we assess both the LLM's ability to produce an initially successful translation, as well as their capacity to fix a previously generated buggy one. If the original and the translated programs are not I/O equivalent, we apply a set of automated feedback strategies, including feedback to the LLM with counterexamples. Our results show that the most successful LLM can translate 47% of our benchmarks, and also provides insights into next steps for improvements. |
| title | Towards Translating Real-World Code with LLMs: A Study of Translating to Rust |
| topic | Software Engineering |
| url | https://arxiv.org/abs/2405.11514 |