Salvato in:
| Autori principali: | , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2506.17335 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912443112882176 |
|---|---|
| author | Yan, Shuo Li, Ruochen Luo, Ziming Wang, Zimu Li, Daoyang Jing, Liqiang He, Kaiyu Wu, Peilin Michalopoulos, George Zhang, Yue Zhang, Ziyang Zhang, Mian Chen, Zhiyu Du, Xinya |
| author_facet | Yan, Shuo Li, Ruochen Luo, Ziming Wang, Zimu Li, Daoyang Jing, Liqiang He, Kaiyu Wu, Peilin Michalopoulos, George Zhang, Yue Zhang, Ziyang Zhang, Mian Chen, Zhiyu Du, Xinya |
| contents | Large language model (LLM) agents have demonstrated remarkable potential in advancing scientific discovery. However, their capability in the fundamental yet crucial task of reproducing code from research papers, especially in the NLP domain, remains underexplored. This task includes unique complex reasoning challenges in the intellectual synthesis of abstract concepts and the comprehension of code repositories with interdependent files. Motivated by this gap, we present LMR-BENCH, a benchmark designed to systematically evaluate the capability of LLM agents on code reproduction from Language Modeling Research. It consists of 28 code reproduction tasks derived from 23 research papers published in top-tier NLP venues over the past five years, spanning nine fundamental categories. Models are provided with a research paper, a code repository containing one or more masked functions, and instructions for implementing these functions. We conduct extensive experiments in standard prompting and LLM agent settings with state-of-the-art LLMs, evaluating the accuracy of unit tests and performing LLM-based evaluation of code correctness. Experimental results reveal that even the most advanced models still exhibit persistent limitations in scientific reasoning and code synthesis, highlighting critical gaps in LLM agents' ability to autonomously reproduce scientific research |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_17335 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | LMR-BENCH: Evaluating LLM Agent's Ability on Reproducing Language Modeling Research Yan, Shuo Li, Ruochen Luo, Ziming Wang, Zimu Li, Daoyang Jing, Liqiang He, Kaiyu Wu, Peilin Michalopoulos, George Zhang, Yue Zhang, Ziyang Zhang, Mian Chen, Zhiyu Du, Xinya Software Engineering Artificial Intelligence Large language model (LLM) agents have demonstrated remarkable potential in advancing scientific discovery. However, their capability in the fundamental yet crucial task of reproducing code from research papers, especially in the NLP domain, remains underexplored. This task includes unique complex reasoning challenges in the intellectual synthesis of abstract concepts and the comprehension of code repositories with interdependent files. Motivated by this gap, we present LMR-BENCH, a benchmark designed to systematically evaluate the capability of LLM agents on code reproduction from Language Modeling Research. It consists of 28 code reproduction tasks derived from 23 research papers published in top-tier NLP venues over the past five years, spanning nine fundamental categories. Models are provided with a research paper, a code repository containing one or more masked functions, and instructions for implementing these functions. We conduct extensive experiments in standard prompting and LLM agent settings with state-of-the-art LLMs, evaluating the accuracy of unit tests and performing LLM-based evaluation of code correctness. Experimental results reveal that even the most advanced models still exhibit persistent limitations in scientific reasoning and code synthesis, highlighting critical gaps in LLM agents' ability to autonomously reproduce scientific research |
| title | LMR-BENCH: Evaluating LLM Agent's Ability on Reproducing Language Modeling Research |
| topic | Software Engineering Artificial Intelligence |
| url | https://arxiv.org/abs/2506.17335 |