ACE-Bench: A Lightweight Benchmark for Evaluating Azure SDK Usage Correctness
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866915931182071808 |
|---|---|
| author | Zhu, Wenxing Qi, Simeng Chen, Junkui Xie, Yan Huang, Min He, Jingkan Wang, Xiao Chen, Cheng Meng, Sijing Zhang, Tianqi |
| author_facet | Zhu, Wenxing Qi, Simeng Chen, Junkui Xie, Yan Huang, Min He, Jingkan Wang, Xiao Chen, Cheng Meng, Sijing Zhang, Tianqi |
| contents | We present ACE-Bench (Azure SDK Coding Evaluation Benchmark), an execution-free benchmark that provides fast, reproducible pass or fail signals for whether large language model (LLM)-based coding agents use Azure SDKs correctly-without provisioning cloud resources or maintaining fragile end-to-end test environments. ACE-Bench turns official Azure SDK documentation examples into self-contained coding tasks and validates solutions with task-specific atomic criteria: deterministic regex checks that enforce required API usage patterns and reference-based LLM-judge checks that capture semantic workflow constraints. This design makes SDK-centric evaluation practical in day-to-day development and CI: it reduces evaluation cost, improves repeatability, and scales to new SDKs and languages as documentation evolves. Using a lightweight coding agent, we benchmark multiple state-of-the-art LLMs and quantify the benefit of retrieval in an MCP-enabled augmented setting, showing consistent gains from documentation access while highlighting substantial cross-model differences. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_09564 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | ACE-Bench: A Lightweight Benchmark for Evaluating Azure SDK Usage Correctness Zhu, Wenxing Qi, Simeng Chen, Junkui Xie, Yan Huang, Min He, Jingkan Wang, Xiao Chen, Cheng Meng, Sijing Zhang, Tianqi Distributed, Parallel, and Cluster Computing Artificial Intelligence Software Engineering I.2 We present ACE-Bench (Azure SDK Coding Evaluation Benchmark), an execution-free benchmark that provides fast, reproducible pass or fail signals for whether large language model (LLM)-based coding agents use Azure SDKs correctly-without provisioning cloud resources or maintaining fragile end-to-end test environments. ACE-Bench turns official Azure SDK documentation examples into self-contained coding tasks and validates solutions with task-specific atomic criteria: deterministic regex checks that enforce required API usage patterns and reference-based LLM-judge checks that capture semantic workflow constraints. This design makes SDK-centric evaluation practical in day-to-day development and CI: it reduces evaluation cost, improves repeatability, and scales to new SDKs and languages as documentation evolves. Using a lightweight coding agent, we benchmark multiple state-of-the-art LLMs and quantify the benefit of retrieval in an MCP-enabled augmented setting, showing consistent gains from documentation access while highlighting substantial cross-model differences. |
| title | ACE-Bench: A Lightweight Benchmark for Evaluating Azure SDK Usage Correctness |
| topic | Distributed, Parallel, and Cluster Computing Artificial Intelligence Software Engineering I.2 |
| url | https://arxiv.org/abs/2604.09564 |