Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2510.06186 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912669060038656 |
|---|---|
| author | Miao, Chunyu Zou, Henry Peng Li, Yangning Chen, Yankai Wang, Yibo Wang, Fangxin Li, Yifan Yang, Wooseong He, Bowei Zhang, Xinni Yu, Dianzhi Yang, Hanchen Nguyen, Hoang H Zhou, Yue Yang, Jie Guo, Jizhou Fan, Wenzhe Yeh, Chin-Yuan Meng, Panpan Fang, Liancheng Qi, Jinhu Huang, Wei-Chieh Gu, Zhengyao Han, Yuwei He, Langzhou Yang, Yuyao Li, Yinghui Zheng, Hai-Tao Liu, Xue King, Irwin Yu, Philip S. |
| author_facet | Miao, Chunyu Zou, Henry Peng Li, Yangning Chen, Yankai Wang, Yibo Wang, Fangxin Li, Yifan Yang, Wooseong He, Bowei Zhang, Xinni Yu, Dianzhi Yang, Hanchen Nguyen, Hoang H Zhou, Yue Yang, Jie Guo, Jizhou Fan, Wenzhe Yeh, Chin-Yuan Meng, Panpan Fang, Liancheng Qi, Jinhu Huang, Wei-Chieh Gu, Zhengyao Han, Yuwei He, Langzhou Yang, Yuyao Li, Yinghui Zheng, Hai-Tao Liu, Xue King, Irwin Yu, Philip S. |
| contents | Large language models (LLMs) show the promise in supporting scientific research implementation, yet their ability to generate correct and executable code remains limited. Existing works largely adopt one-shot settings, ignoring the iterative and feedback-driven nature of realistic workflows of scientific research development. To address this gap, we present RECODE-H, a benchmark of 102 tasks from research papers and repositories that evaluates LLM agents through multi-turn interactions with LLM-simulated human feedback. It includes structured instructions,unit tests, and a five-level feedback hierarchy to reflect realistic researcher-agent collaboration. We further present ReCodeAgent, a framework that integrates feedback into iterative code generation. Experiments with leading LLMs, including GPT-5, Claude-Sonnet-4, DeepSeek-V3.1, and Gemini 2.5, show substantial performance gains with richer feedback, while also highlighting ongoing challenges in the generation of complex research code. RECODE-H establishes a foundation for developing adaptive, feedback-driven LLM agents in scientific research implementation |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_06186 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | RECODE-H: A Benchmark for Research Code Development with Interactive Human Feedback Miao, Chunyu Zou, Henry Peng Li, Yangning Chen, Yankai Wang, Yibo Wang, Fangxin Li, Yifan Yang, Wooseong He, Bowei Zhang, Xinni Yu, Dianzhi Yang, Hanchen Nguyen, Hoang H Zhou, Yue Yang, Jie Guo, Jizhou Fan, Wenzhe Yeh, Chin-Yuan Meng, Panpan Fang, Liancheng Qi, Jinhu Huang, Wei-Chieh Gu, Zhengyao Han, Yuwei He, Langzhou Yang, Yuyao Li, Yinghui Zheng, Hai-Tao Liu, Xue King, Irwin Yu, Philip S. Computation and Language Artificial Intelligence Large language models (LLMs) show the promise in supporting scientific research implementation, yet their ability to generate correct and executable code remains limited. Existing works largely adopt one-shot settings, ignoring the iterative and feedback-driven nature of realistic workflows of scientific research development. To address this gap, we present RECODE-H, a benchmark of 102 tasks from research papers and repositories that evaluates LLM agents through multi-turn interactions with LLM-simulated human feedback. It includes structured instructions,unit tests, and a five-level feedback hierarchy to reflect realistic researcher-agent collaboration. We further present ReCodeAgent, a framework that integrates feedback into iterative code generation. Experiments with leading LLMs, including GPT-5, Claude-Sonnet-4, DeepSeek-V3.1, and Gemini 2.5, show substantial performance gains with richer feedback, while also highlighting ongoing challenges in the generation of complex research code. RECODE-H establishes a foundation for developing adaptive, feedback-driven LLM agents in scientific research implementation |
| title | RECODE-H: A Benchmark for Research Code Development with Interactive Human Feedback |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2510.06186 |