SWE-Sharp-Bench: A Reproducible Benchmark for C# Software Engineering Tasks
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866912716005834752 |
|---|---|
| author | Mhatre, Sanket Bajpai, Yasharth Gulwani, Sumit Murphy-Hill, Emerson Soares, Gustavo |
| author_facet | Mhatre, Sanket Bajpai, Yasharth Gulwani, Sumit Murphy-Hill, Emerson Soares, Gustavo |
| contents | AI coding agents have shown great progress on Python software engineering benchmarks like SWE-Bench, and for other languages like Java and C in benchmarks like Multi-SWE-Bench. However, C# -- a prominent enterprise language ranking #5 in the TIOBE index -- remains absent from such benchmarks. We introduce SWE-Sharp-Bench, a reproducible software engineering benchmark for C# featuring 150 instances from 17 repositories. Evaluating identical model-agent configurations across languages reveals a significant performance gap: while 70% of Python tasks in SWE-Bench Verified are solved, only 40% of our C# tasks are resolved. We open-source SWE-Sharp-Bench and our entire curation pipeline. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_02352 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SWE-Sharp-Bench: A Reproducible Benchmark for C# Software Engineering Tasks Mhatre, Sanket Bajpai, Yasharth Gulwani, Sumit Murphy-Hill, Emerson Soares, Gustavo Software Engineering AI coding agents have shown great progress on Python software engineering benchmarks like SWE-Bench, and for other languages like Java and C in benchmarks like Multi-SWE-Bench. However, C# -- a prominent enterprise language ranking #5 in the TIOBE index -- remains absent from such benchmarks. We introduce SWE-Sharp-Bench, a reproducible software engineering benchmark for C# featuring 150 instances from 17 repositories. Evaluating identical model-agent configurations across languages reveals a significant performance gap: while 70% of Python tasks in SWE-Bench Verified are solved, only 40% of our C# tasks are resolved. We open-source SWE-Sharp-Bench and our entire curation pipeline. |
| title | SWE-Sharp-Bench: A Reproducible Benchmark for C# Software Engineering Tasks |
| topic | Software Engineering |
| url | https://arxiv.org/abs/2511.02352 |