DI-BENCH: Benchmarking Large Language Models on Dependency Inference with Testable Repositories at Scale
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910796027527168 |
|---|---|
| author | Zhang, Linghao Wang, Junhao He, Shilin Zhang, Chaoyun Kang, Yu Li, Bowen Wen, Jiaheng Xie, Chengxing Wang, Maoquan Huang, Yufan Nallipogu, Elsie Lin, Qingwei Dang, Yingnong Rajmohan, Saravan Zhang, Dongmei Zhang, Qi |
| author_facet | Zhang, Linghao Wang, Junhao He, Shilin Zhang, Chaoyun Kang, Yu Li, Bowen Wen, Jiaheng Xie, Chengxing Wang, Maoquan Huang, Yufan Nallipogu, Elsie Lin, Qingwei Dang, Yingnong Rajmohan, Saravan Zhang, Dongmei Zhang, Qi |
| contents | Large Language Models have advanced automated software development, however, it remains a challenge to correctly infer dependencies, namely, identifying the internal components and external packages required for a repository to successfully run. Existing studies highlight that dependency-related issues cause over 40\% of observed runtime errors on the generated repository. To address this, we introduce DI-BENCH, a large-scale benchmark and evaluation framework specifically designed to assess LLMs' capability on dependency inference. The benchmark features 581 repositories with testing environments across Python, C#, Rust, and JavaScript. Extensive experiments with textual and execution-based metrics reveal that the current best-performing model achieves only a 42.9% execution pass rate, indicating significant room for improvement. DI-BENCH establishes a new viewpoint for evaluating LLM performance on repositories, paving the way for more robust end-to-end software synthesis. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_13699 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | DI-BENCH: Benchmarking Large Language Models on Dependency Inference with Testable Repositories at Scale Zhang, Linghao Wang, Junhao He, Shilin Zhang, Chaoyun Kang, Yu Li, Bowen Wen, Jiaheng Xie, Chengxing Wang, Maoquan Huang, Yufan Nallipogu, Elsie Lin, Qingwei Dang, Yingnong Rajmohan, Saravan Zhang, Dongmei Zhang, Qi Computation and Language Software Engineering Large Language Models have advanced automated software development, however, it remains a challenge to correctly infer dependencies, namely, identifying the internal components and external packages required for a repository to successfully run. Existing studies highlight that dependency-related issues cause over 40\% of observed runtime errors on the generated repository. To address this, we introduce DI-BENCH, a large-scale benchmark and evaluation framework specifically designed to assess LLMs' capability on dependency inference. The benchmark features 581 repositories with testing environments across Python, C#, Rust, and JavaScript. Extensive experiments with textual and execution-based metrics reveal that the current best-performing model achieves only a 42.9% execution pass rate, indicating significant room for improvement. DI-BENCH establishes a new viewpoint for evaluating LLM performance on repositories, paving the way for more robust end-to-end software synthesis. |
| title | DI-BENCH: Benchmarking Large Language Models on Dependency Inference with Testable Repositories at Scale |
| topic | Computation and Language Software Engineering |
| url | https://arxiv.org/abs/2501.13699 |