TransLibEval: Demystify Large Language Models' Capability in Third-party Library-targeted Code Translation
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911381293367296 |
|---|---|
| author | Xue, Pengyu Zheng, Kunwu Yang, Zhen Pei, Yifei Wu, Linhao Dong, Jiahui Luo, Xiapu Xiao, Yan Liu, Fei Zhang, Yuxuan Lyu, Xiran Li, Xianhang Zhu, Xuanyu Wang, Chengyi |
| author_facet | Xue, Pengyu Zheng, Kunwu Yang, Zhen Pei, Yifei Wu, Linhao Dong, Jiahui Luo, Xiapu Xiao, Yan Liu, Fei Zhang, Yuxuan Lyu, Xiran Li, Xianhang Zhu, Xuanyu Wang, Chengyi |
| contents | In recent years, Large Language Models (LLMs) have been widely studied in the code translation field on the method, class, and even repository levels. However, most of these benchmarks are limited in terms of Third-Party Library (TPL) categories and scales, making TPL-related errors hard to expose and hindering the development of targeted solutions. Considering the high dependence (over 90%) on TPLs in practical programming, demystifying and analyzing LLMs' code translation performance involving various TPLs becomes imperative. To address this gap, we construct TransLibEval, the first benchmark dedicated to library-centric code translation. It consists of 200 real-world tasks across Python, Java, and C++, each explicitly involving TPLs from diverse categories such as data processing, machine learning, and web development, with comprehensive dependency coverage and high-coverage test suites. We evaluate seven recent LLMs of commercial, general, and code-specialized families under six translation strategies of three categories: Direct, IR-guided, and Retrieval-augmented. Experimental results show a dramatic performance drop compared with library-free settings (average CA decline over 60%), while diverse strategies demonstrate heterogeneous advantages. Furthermore, we analyze 4,831 failed cases from GPT-4o, one of the State-of-the-Art (SOTA) LLMs, revealing numerous third-party reference errors that were obscured previously. These findings highlight the unique challenges of library-centric translation and provide practical guidance for improving TPL-aware code intelligence. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_12087 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | TransLibEval: Demystify Large Language Models' Capability in Third-party Library-targeted Code Translation Xue, Pengyu Zheng, Kunwu Yang, Zhen Pei, Yifei Wu, Linhao Dong, Jiahui Luo, Xiapu Xiao, Yan Liu, Fei Zhang, Yuxuan Lyu, Xiran Li, Xianhang Zhu, Xuanyu Wang, Chengyi Software Engineering In recent years, Large Language Models (LLMs) have been widely studied in the code translation field on the method, class, and even repository levels. However, most of these benchmarks are limited in terms of Third-Party Library (TPL) categories and scales, making TPL-related errors hard to expose and hindering the development of targeted solutions. Considering the high dependence (over 90%) on TPLs in practical programming, demystifying and analyzing LLMs' code translation performance involving various TPLs becomes imperative. To address this gap, we construct TransLibEval, the first benchmark dedicated to library-centric code translation. It consists of 200 real-world tasks across Python, Java, and C++, each explicitly involving TPLs from diverse categories such as data processing, machine learning, and web development, with comprehensive dependency coverage and high-coverage test suites. We evaluate seven recent LLMs of commercial, general, and code-specialized families under six translation strategies of three categories: Direct, IR-guided, and Retrieval-augmented. Experimental results show a dramatic performance drop compared with library-free settings (average CA decline over 60%), while diverse strategies demonstrate heterogeneous advantages. Furthermore, we analyze 4,831 failed cases from GPT-4o, one of the State-of-the-Art (SOTA) LLMs, revealing numerous third-party reference errors that were obscured previously. These findings highlight the unique challenges of library-centric translation and provide practical guidance for improving TPL-aware code intelligence. |
| title | TransLibEval: Demystify Large Language Models' Capability in Third-party Library-targeted Code Translation |
| topic | Software Engineering |
| url | https://arxiv.org/abs/2509.12087 |