SwingArena: Competitive Programming Arena for Long-context GitHub Issue Solving
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915840650117120 |
|---|---|
| author | Xu, Wendong Xiong, Jing Zhao, Chenyang Chen, Qiujiang Wang, Haoran Shen, Hui Wan, Zhongwei Dai, Jianbo Wu, Taiqiang Xiao, He Tao, Chaofan Mao, Z. Morley Sheng, Ying Guo, Zhijiang Yang, Hongxia Yu, Bei Kong, Lingpeng Gu, Quanquan Wong, Ngai |
| author_facet | Xu, Wendong Xiong, Jing Zhao, Chenyang Chen, Qiujiang Wang, Haoran Shen, Hui Wan, Zhongwei Dai, Jianbo Wu, Taiqiang Xiao, He Tao, Chaofan Mao, Z. Morley Sheng, Ying Guo, Zhijiang Yang, Hongxia Yu, Bei Kong, Lingpeng Gu, Quanquan Wong, Ngai |
| contents | We present SwingArena, a competitive evaluation framework for Large Language Models (LLMs) that closely mirrors real-world software development workflows. Unlike traditional static benchmarks, SwingArena models the collaborative process of software iteration by pairing LLMs as submitters, who generate patches, and reviewers, who create test cases and verify the patches through continuous integration (CI) pipelines. To support these interactive evaluations, we introduce a retrieval-augmented code generation (RACG) module that efficiently handles long-context challenges by providing syntactically and semantically relevant code snippets from large codebases, supporting multiple programming languages (C++, Python, Rust, and Go). This enables the framework to scale across diverse tasks and contexts while respecting token limitations. Our experiments, using over 400 high-quality real-world GitHub issues selected from a pool of 2,300 issues, show that models like GPT-4o excel at aggressive patch generation, whereas DeepSeek and Gemini prioritize correctness in CI validation. SwingArena presents a scalable and extensible methodology for evaluating LLMs in realistic, CI-driven software development settings. More details are available on our project page: swing-bench.github.io |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_23932 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SwingArena: Competitive Programming Arena for Long-context GitHub Issue Solving Xu, Wendong Xiong, Jing Zhao, Chenyang Chen, Qiujiang Wang, Haoran Shen, Hui Wan, Zhongwei Dai, Jianbo Wu, Taiqiang Xiao, He Tao, Chaofan Mao, Z. Morley Sheng, Ying Guo, Zhijiang Yang, Hongxia Yu, Bei Kong, Lingpeng Gu, Quanquan Wong, Ngai Computation and Language We present SwingArena, a competitive evaluation framework for Large Language Models (LLMs) that closely mirrors real-world software development workflows. Unlike traditional static benchmarks, SwingArena models the collaborative process of software iteration by pairing LLMs as submitters, who generate patches, and reviewers, who create test cases and verify the patches through continuous integration (CI) pipelines. To support these interactive evaluations, we introduce a retrieval-augmented code generation (RACG) module that efficiently handles long-context challenges by providing syntactically and semantically relevant code snippets from large codebases, supporting multiple programming languages (C++, Python, Rust, and Go). This enables the framework to scale across diverse tasks and contexts while respecting token limitations. Our experiments, using over 400 high-quality real-world GitHub issues selected from a pool of 2,300 issues, show that models like GPT-4o excel at aggressive patch generation, whereas DeepSeek and Gemini prioritize correctness in CI validation. SwingArena presents a scalable and extensible methodology for evaluating LLMs in realistic, CI-driven software development settings. More details are available on our project page: swing-bench.github.io |
| title | SwingArena: Competitive Programming Arena for Long-context GitHub Issue Solving |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2505.23932 |