Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2508.09791 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908487846461440 |
|---|---|
| author | Han, Junxiao Wang, Yarong Gu, Xiaodong Gao, Cuiyun Wan, Yao Han, Song Lo, David Deng, Shuiguang |
| author_facet | Han, Junxiao Wang, Yarong Gu, Xiaodong Gao, Cuiyun Wan, Yao Han, Song Lo, David Deng, Shuiguang |
| contents | In this paper, we propose LibRec, a novel framework that integrates the capabilities of LLMs with retrieval-augmented generation(RAG) techniques to automate the recommendation of alternative libraries. The framework further employs in-context learning to extract migration intents from commit messages to enhance the accuracy of its recommendations. To evaluate the effectiveness of LibRec, we introduce LibEval, a benchmark designed to assess the performance in the library migration recommendation task. LibEval comprises 2,888 migration records associated with 2,368 libraries extracted from 2,324 Python repositories. Each migration record captures source-target library pairs, along with their corresponding migration intents and intent types. Based on LibEval, we evaluated the effectiveness of ten popular LLMs within our framework, conducted an ablation study to examine the contributions of key components within our framework, explored the impact of various prompt strategies on the framework's performance, assessed its effectiveness across various intent types, and performed detailed failure case analyses. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_09791 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | LibRec: Benchmarking Retrieval-Augmented LLMs for Library Migration Recommendations Han, Junxiao Wang, Yarong Gu, Xiaodong Gao, Cuiyun Wan, Yao Han, Song Lo, David Deng, Shuiguang Software Engineering Artificial Intelligence In this paper, we propose LibRec, a novel framework that integrates the capabilities of LLMs with retrieval-augmented generation(RAG) techniques to automate the recommendation of alternative libraries. The framework further employs in-context learning to extract migration intents from commit messages to enhance the accuracy of its recommendations. To evaluate the effectiveness of LibRec, we introduce LibEval, a benchmark designed to assess the performance in the library migration recommendation task. LibEval comprises 2,888 migration records associated with 2,368 libraries extracted from 2,324 Python repositories. Each migration record captures source-target library pairs, along with their corresponding migration intents and intent types. Based on LibEval, we evaluated the effectiveness of ten popular LLMs within our framework, conducted an ablation study to examine the contributions of key components within our framework, explored the impact of various prompt strategies on the framework's performance, assessed its effectiveness across various intent types, and performed detailed failure case analyses. |
| title | LibRec: Benchmarking Retrieval-Augmented LLMs for Library Migration Recommendations |
| topic | Software Engineering Artificial Intelligence |
| url | https://arxiv.org/abs/2508.09791 |