Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2604.12162 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917406094393344 |
|---|---|
| author | Lu, Pengrui Xu, Bingyu Zhang, Wenjun Hua, Shengjia Gao, Xuanjian Ge, Ranxiang Ye, Lyumanshan Wu, Linxuan Li, Yiran Yu, Junfei Fish Zhang, Yibo Li, Ruixin Li, Manxiang Han, Xiao Zhou, Xiaocong Chi, Guangyao Chen, Zisheng Chen, Kaishen Wang, Kun Xu, Qihua Meng, Fengyue Ni, Yuchen Li, Jiajun Liu, Jinxiu Zhang, Danfeng Zhao, Jingru Liu, Pengfei |
| author_facet | Lu, Pengrui Xu, Bingyu Zhang, Wenjun Hua, Shengjia Gao, Xuanjian Ge, Ranxiang Ye, Lyumanshan Wu, Linxuan Li, Yiran Yu, Junfei Fish Zhang, Yibo Li, Ruixin Li, Manxiang Han, Xiao Zhou, Xiaocong Chi, Guangyao Chen, Zisheng Chen, Kaishen Wang, Kun Xu, Qihua Meng, Fengyue Ni, Yuchen Li, Jiajun Liu, Jinxiu Zhang, Danfeng Zhao, Jingru Liu, Pengfei |
| contents | The rapid deployment of AI agents in commercial settings has outpaced the development of evaluation methodologies that reflect production realities. Existing benchmarks measure agent capabilities through retrospectively curated tasks with well-specified requirements and deterministic metrics -- conditions that diverge fundamentally from production environments where requirements contain implicit constraints, inputs are heterogeneous multi-modal documents with information fragmented across sources, tasks demand undeclared domain expertise, outputs are long-horizon professional deliverables, and success is judged by domain experts whose standards evolve over time. We present AlphaEval, a production-grounded benchmark of 94 tasks sourced from seven companies deploying AI agents in their core business, spanning six O*NET (Occupational Information Network) domains. Unlike model-centric benchmarks, AlphaEval evaluates complete agent products -- Claude Code, Codex, etc. -- as commercial systems, capturing performance variations invisible to model-level evaluation. Our evaluation framework covers multiple paradigms (LLM-as-a-Judge, reference-driven metrics, formal verification, rubric-based assessment, automated UI testing, etc.), with individual domains composing multiple paradigms. Beyond the benchmark itself, we contribute a requirement-to-benchmark construction framework -- a systematic methodology that transforms authentic production requirements into executable evaluation tasks in minimal time. This framework standardizes the entire pipeline from requirement to evaluation, providing a reproducible, modular process that any organization can adopt to construct production-grounded benchmarks for their own domains. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_12162 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | AlphaEval: Evaluating Agents in Production Lu, Pengrui Xu, Bingyu Zhang, Wenjun Hua, Shengjia Gao, Xuanjian Ge, Ranxiang Ye, Lyumanshan Wu, Linxuan Li, Yiran Yu, Junfei Fish Zhang, Yibo Li, Ruixin Li, Manxiang Han, Xiao Zhou, Xiaocong Chi, Guangyao Chen, Zisheng Chen, Kaishen Wang, Kun Xu, Qihua Meng, Fengyue Ni, Yuchen Li, Jiajun Liu, Jinxiu Zhang, Danfeng Zhao, Jingru Liu, Pengfei Computation and Language The rapid deployment of AI agents in commercial settings has outpaced the development of evaluation methodologies that reflect production realities. Existing benchmarks measure agent capabilities through retrospectively curated tasks with well-specified requirements and deterministic metrics -- conditions that diverge fundamentally from production environments where requirements contain implicit constraints, inputs are heterogeneous multi-modal documents with information fragmented across sources, tasks demand undeclared domain expertise, outputs are long-horizon professional deliverables, and success is judged by domain experts whose standards evolve over time. We present AlphaEval, a production-grounded benchmark of 94 tasks sourced from seven companies deploying AI agents in their core business, spanning six O*NET (Occupational Information Network) domains. Unlike model-centric benchmarks, AlphaEval evaluates complete agent products -- Claude Code, Codex, etc. -- as commercial systems, capturing performance variations invisible to model-level evaluation. Our evaluation framework covers multiple paradigms (LLM-as-a-Judge, reference-driven metrics, formal verification, rubric-based assessment, automated UI testing, etc.), with individual domains composing multiple paradigms. Beyond the benchmark itself, we contribute a requirement-to-benchmark construction framework -- a systematic methodology that transforms authentic production requirements into executable evaluation tasks in minimal time. This framework standardizes the entire pipeline from requirement to evaluation, providing a reproducible, modular process that any organization can adopt to construct production-grounded benchmarks for their own domains. |
| title | AlphaEval: Evaluating Agents in Production |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2604.12162 |