SWE-Compass: Towards Unified Evaluation of Agentic Coding Abilities for Large Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912700859154432 |
|---|---|
| author | Xu, Jingxuan Deng, Ken Li, Weihao Yu, Songwei Tang, Huaixi Huang, Haoyang Lai, Zhiyi Zhan, Zizheng Wu, Yanan Zhang, Chenchen Lei, Kepeng Yao, Yifan Lei, Xinping Zhu, Wenqiang Feng, Zongxian Li, Han Xiong, Junqi Li, Dailin Gao, Zuchen Wu, Kun Xiang, Wen Zhan, Ziqi Zhang, Yuanxing Gong, Wuxuan Gao, Ziyuan Wang, Guanxiang Xue, Yirong Li, Mengtong Xie, Mengfei Zhang, Xiaojiang Wang, Jinghui Zhuang, Wenhao Lin, Zheng Wang, Huiming Zhang, Zhaoxiang Zhang, Yuqun Zhang, Haotian Chen, Bin Liu, Jiaheng |
| author_facet | Xu, Jingxuan Deng, Ken Li, Weihao Yu, Songwei Tang, Huaixi Huang, Haoyang Lai, Zhiyi Zhan, Zizheng Wu, Yanan Zhang, Chenchen Lei, Kepeng Yao, Yifan Lei, Xinping Zhu, Wenqiang Feng, Zongxian Li, Han Xiong, Junqi Li, Dailin Gao, Zuchen Wu, Kun Xiang, Wen Zhan, Ziqi Zhang, Yuanxing Gong, Wuxuan Gao, Ziyuan Wang, Guanxiang Xue, Yirong Li, Mengtong Xie, Mengfei Zhang, Xiaojiang Wang, Jinghui Zhuang, Wenhao Lin, Zheng Wang, Huiming Zhang, Zhaoxiang Zhang, Yuqun Zhang, Haotian Chen, Bin Liu, Jiaheng |
| contents | Evaluating large language models (LLMs) for software engineering has been limited by narrow task coverage, language bias, and insufficient alignment with real-world developer workflows. Existing benchmarks often focus on algorithmic problems or Python-centric bug fixing, leaving critical dimensions of software engineering underexplored. To address these gaps, we introduce SWE-Compass1, a comprehensive benchmark that unifies heterogeneous code-related evaluations into a structured and production-aligned framework. SWE-Compass spans 8 task types, 8 programming scenarios, and 10 programming languages, with 2000 high-quality instances curated from authentic GitHub pull requests and refined through systematic filtering and validation. We benchmark ten state-of-the-art LLMs under two agentic frameworks, SWE-Agent and Claude Code, revealing a clear hierarchy of difficulty across task types, languages, and scenarios. Moreover, by aligning evaluation with real-world developer practices, SWE-Compass provides a rigorous and reproducible foundation for diagnosing and advancing agentic coding capabilities in large language models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_05459 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SWE-Compass: Towards Unified Evaluation of Agentic Coding Abilities for Large Language Models Xu, Jingxuan Deng, Ken Li, Weihao Yu, Songwei Tang, Huaixi Huang, Haoyang Lai, Zhiyi Zhan, Zizheng Wu, Yanan Zhang, Chenchen Lei, Kepeng Yao, Yifan Lei, Xinping Zhu, Wenqiang Feng, Zongxian Li, Han Xiong, Junqi Li, Dailin Gao, Zuchen Wu, Kun Xiang, Wen Zhan, Ziqi Zhang, Yuanxing Gong, Wuxuan Gao, Ziyuan Wang, Guanxiang Xue, Yirong Li, Mengtong Xie, Mengfei Zhang, Xiaojiang Wang, Jinghui Zhuang, Wenhao Lin, Zheng Wang, Huiming Zhang, Zhaoxiang Zhang, Yuqun Zhang, Haotian Chen, Bin Liu, Jiaheng Software Engineering Artificial Intelligence Evaluating large language models (LLMs) for software engineering has been limited by narrow task coverage, language bias, and insufficient alignment with real-world developer workflows. Existing benchmarks often focus on algorithmic problems or Python-centric bug fixing, leaving critical dimensions of software engineering underexplored. To address these gaps, we introduce SWE-Compass1, a comprehensive benchmark that unifies heterogeneous code-related evaluations into a structured and production-aligned framework. SWE-Compass spans 8 task types, 8 programming scenarios, and 10 programming languages, with 2000 high-quality instances curated from authentic GitHub pull requests and refined through systematic filtering and validation. We benchmark ten state-of-the-art LLMs under two agentic frameworks, SWE-Agent and Claude Code, revealing a clear hierarchy of difficulty across task types, languages, and scenarios. Moreover, by aligning evaluation with real-world developer practices, SWE-Compass provides a rigorous and reproducible foundation for diagnosing and advancing agentic coding capabilities in large language models. |
| title | SWE-Compass: Towards Unified Evaluation of Agentic Coding Abilities for Large Language Models |
| topic | Software Engineering Artificial Intelligence |
| url | https://arxiv.org/abs/2511.05459 |