Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2603.07980 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908873128935424 |
|---|---|
| author | Yang, Qianyu Liu, Yang Li, Jiaqi Bai, Jun Chen, Hao Chen, Kaiyuan Duan, Tiliang Dong, Jiayun Hu, Xiaobo Jia, Zixia Liu, Yang Peng, Tao Ren, Yixin Tian, Ran Wang, Zaiyuan Xiao, Yanglihong Yao, Gang Yin, Lingyue Zhang, Ge Zhang, Chun Jiao, Jianpeng Zheng, Zilong Gong, Yuan |
| author_facet | Yang, Qianyu Liu, Yang Li, Jiaqi Bai, Jun Chen, Hao Chen, Kaiyuan Duan, Tiliang Dong, Jiayun Hu, Xiaobo Jia, Zixia Liu, Yang Peng, Tao Ren, Yixin Tian, Ran Wang, Zaiyuan Xiao, Yanglihong Yao, Gang Yin, Lingyue Zhang, Ge Zhang, Chun Jiao, Jianpeng Zheng, Zilong Gong, Yuan |
| contents | As language models (LMs) evolve from chat assistants to long-horizon agents capable of multi-step reasoning and tool use, existing benchmarks remain largely confined to structured or exam-style tasks that fall short of real-world professional demands. To this end, we introduce \$OneMillion-Bench \$OneMillion-Bench, a benchmark of 400 expert-curated tasks spanning Law, Finance, Industry, Healthcare, and Natural Science, built to evaluate agents across economically consequential scenarios. Unlike prior work, the benchmark requires retrieving authoritative sources, resolving conflicting evidence, applying domain-specific rules, and making constraint decisions, where correctness depends as much on the reasoning process as the final answer. We adopt a rubric-based evaluation protocol scoring factual accuracy, logical coherence, practical feasibility, and professional compliance, focused on expert-level problems to ensure meaningful differentiation across agents. Together, \$OneMillion-Bench provides a unified testbed for assessing agentic reliability, professional depth, and practical readiness in domain-intensive scenarios. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_07980 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | \$OneMillion-Bench: How Far are Language Agents from Human Experts? Yang, Qianyu Liu, Yang Li, Jiaqi Bai, Jun Chen, Hao Chen, Kaiyuan Duan, Tiliang Dong, Jiayun Hu, Xiaobo Jia, Zixia Liu, Yang Peng, Tao Ren, Yixin Tian, Ran Wang, Zaiyuan Xiao, Yanglihong Yao, Gang Yin, Lingyue Zhang, Ge Zhang, Chun Jiao, Jianpeng Zheng, Zilong Gong, Yuan Machine Learning Artificial Intelligence Computation and Language As language models (LMs) evolve from chat assistants to long-horizon agents capable of multi-step reasoning and tool use, existing benchmarks remain largely confined to structured or exam-style tasks that fall short of real-world professional demands. To this end, we introduce \$OneMillion-Bench \$OneMillion-Bench, a benchmark of 400 expert-curated tasks spanning Law, Finance, Industry, Healthcare, and Natural Science, built to evaluate agents across economically consequential scenarios. Unlike prior work, the benchmark requires retrieving authoritative sources, resolving conflicting evidence, applying domain-specific rules, and making constraint decisions, where correctness depends as much on the reasoning process as the final answer. We adopt a rubric-based evaluation protocol scoring factual accuracy, logical coherence, practical feasibility, and professional compliance, focused on expert-level problems to ensure meaningful differentiation across agents. Together, \$OneMillion-Bench provides a unified testbed for assessing agentic reliability, professional depth, and practical readiness in domain-intensive scenarios. |
| title | \$OneMillion-Bench: How Far are Language Agents from Human Experts? |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2603.07980 |