Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2505.16667 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918030465826816 |
|---|---|
| author | Yang, Xinwei Liu, Zhaofeng Huang, Chen Zhang, Jiashuai Zhang, Tong Zhang, Yifan Lei, Wenqiang |
| author_facet | Yang, Xinwei Liu, Zhaofeng Huang, Chen Zhang, Jiashuai Zhang, Tong Zhang, Yifan Lei, Wenqiang |
| contents | While recent research increasingly emphasizes the value of human-LLM collaboration in competitive programming and proposes numerous empirical methods, a comprehensive understanding remains elusive due to the fragmented nature of existing studies and their use of diverse, application-specific human feedback. Thus, our work serves a three-fold purpose: First, we present the first taxonomy of human feedback consolidating the entire programming process, which promotes fine-grained evaluation. Second, we introduce ELABORATIONSET, a novel programming dataset specifically designed for human-LLM collaboration, meticulously annotated to enable large-scale simulated human feedback and facilitate costeffective real human interaction studies. Third, we introduce ELABORATION, a novel benchmark to facilitate a thorough assessment of human-LLM competitive programming. With ELABORATION, we pinpoint strengthes and weaknesses of existing methods, thereby setting the foundation for future improvement. Our code and dataset are available at https://github.com/SCUNLP/ELABORATION |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_16667 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ELABORATION: A Comprehensive Benchmark on Human-LLM Competitive Programming Yang, Xinwei Liu, Zhaofeng Huang, Chen Zhang, Jiashuai Zhang, Tong Zhang, Yifan Lei, Wenqiang Artificial Intelligence While recent research increasingly emphasizes the value of human-LLM collaboration in competitive programming and proposes numerous empirical methods, a comprehensive understanding remains elusive due to the fragmented nature of existing studies and their use of diverse, application-specific human feedback. Thus, our work serves a three-fold purpose: First, we present the first taxonomy of human feedback consolidating the entire programming process, which promotes fine-grained evaluation. Second, we introduce ELABORATIONSET, a novel programming dataset specifically designed for human-LLM collaboration, meticulously annotated to enable large-scale simulated human feedback and facilitate costeffective real human interaction studies. Third, we introduce ELABORATION, a novel benchmark to facilitate a thorough assessment of human-LLM competitive programming. With ELABORATION, we pinpoint strengthes and weaknesses of existing methods, thereby setting the foundation for future improvement. Our code and dataset are available at https://github.com/SCUNLP/ELABORATION |
| title | ELABORATION: A Comprehensive Benchmark on Human-LLM Competitive Programming |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2505.16667 |