Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2604.02368 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917424514727936 |
|---|---|
| author | Liu, Xue Ma, Xin Ma, Yuxin Peng, Yongchang Wang, Duo Wen, Zhoufutu Zhang, Ge Zhang, Kaiyuan Chen, Xinyu Ding, Yida He, Tianci Hou, Jiani Hu, Liang Huang, Ziyun Hui, Yongzhe Jiao, Jianpeng Ju, Chennan Kong, Yingru Li, Yiran Liu, Jiashuo Liu, Mengyun Ma, Luyao Ni, Fei Ni, Yiqing Niu, Pengbo Qiu, Yueyan Ren, Yanle Shen, Xinyu Shi, Zilin Wang, Zaiyuan Yue, Wenjie Zhang, Chun Zhang, Shiyu Zhang, Xinyi Zhao, Kaiwen Zhu, Zhenwei Wu, Shanshan Zhao, Qi Huang, Wenhao |
| author_facet | Liu, Xue Ma, Xin Ma, Yuxin Peng, Yongchang Wang, Duo Wen, Zhoufutu Zhang, Ge Zhang, Kaiyuan Chen, Xinyu Ding, Yida He, Tianci Hou, Jiani Hu, Liang Huang, Ziyun Hui, Yongzhe Jiao, Jianpeng Ju, Chennan Kong, Yingru Li, Yiran Liu, Jiashuo Liu, Mengyun Ma, Luyao Ni, Fei Ni, Yiqing Niu, Pengbo Qiu, Yueyan Ren, Yanle Shen, Xinyu Shi, Zilin Wang, Zaiyuan Yue, Wenjie Zhang, Chun Zhang, Shiyu Zhang, Xinyi Zhao, Kaiwen Zhu, Zhenwei Wu, Shanshan Zhao, Qi Huang, Wenhao |
| contents | As Large Language Models (LLMs) exhibit plateauing performance on conventional benchmarks, a pivotal challenge persists: evaluating their proficiency in complex, open-ended tasks characterizing genuine expert-level cognition. Existing frameworks suffer from narrow domain coverage, reliance on generalist tasks, or self-evaluation biases. To bridge this gap, we present XpertBench, a high-fidelity benchmark engineered to assess LLMs across authentic professional domains. XpertBench consists of 1,346 meticulously curated tasks across 80 categories, spanning finance, healthcare, legal services, education, and dual-track research (STEM and Humanities). These tasks are derived from over 1,000 submissions by domain experts--including researchers from elite institutions and practitioners with extensive clinical or industrial experience--ensuring superior ecological validity. Each task uses detailed rubrics with mostly 15-40 weighted checkpoints to assess professional rigor. To facilitate scalable yet human-aligned assessment, we introduce ShotJudge, a novel evaluation paradigm that employs LLM judges calibrated with expert few-shot exemplars to mitigate self-rewarding biases. Our empirical evaluation of state-of-the-art LLMs reveals a pronounced performance ceiling: even leading models achieve a peak success rate of only ~66%, with a mean score around 55%. Models also exhibit domain-specific divergence, showing non-overlapping strengths in quantitative reasoning versus linguistic synthesis.. These findings underscore a significant "expert-gap" in current AI systems and establish XpertBench as a critical instrument for navigating the transition from general-purpose assistants to specialized professional collaborators. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_02368 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Xpertbench: Expert Level Tasks with Rubrics-Based Evaluation Liu, Xue Ma, Xin Ma, Yuxin Peng, Yongchang Wang, Duo Wen, Zhoufutu Zhang, Ge Zhang, Kaiyuan Chen, Xinyu Ding, Yida He, Tianci Hou, Jiani Hu, Liang Huang, Ziyun Hui, Yongzhe Jiao, Jianpeng Ju, Chennan Kong, Yingru Li, Yiran Liu, Jiashuo Liu, Mengyun Ma, Luyao Ni, Fei Ni, Yiqing Niu, Pengbo Qiu, Yueyan Ren, Yanle Shen, Xinyu Shi, Zilin Wang, Zaiyuan Yue, Wenjie Zhang, Chun Zhang, Shiyu Zhang, Xinyi Zhao, Kaiwen Zhu, Zhenwei Wu, Shanshan Zhao, Qi Huang, Wenhao Artificial Intelligence Computation and Language As Large Language Models (LLMs) exhibit plateauing performance on conventional benchmarks, a pivotal challenge persists: evaluating their proficiency in complex, open-ended tasks characterizing genuine expert-level cognition. Existing frameworks suffer from narrow domain coverage, reliance on generalist tasks, or self-evaluation biases. To bridge this gap, we present XpertBench, a high-fidelity benchmark engineered to assess LLMs across authentic professional domains. XpertBench consists of 1,346 meticulously curated tasks across 80 categories, spanning finance, healthcare, legal services, education, and dual-track research (STEM and Humanities). These tasks are derived from over 1,000 submissions by domain experts--including researchers from elite institutions and practitioners with extensive clinical or industrial experience--ensuring superior ecological validity. Each task uses detailed rubrics with mostly 15-40 weighted checkpoints to assess professional rigor. To facilitate scalable yet human-aligned assessment, we introduce ShotJudge, a novel evaluation paradigm that employs LLM judges calibrated with expert few-shot exemplars to mitigate self-rewarding biases. Our empirical evaluation of state-of-the-art LLMs reveals a pronounced performance ceiling: even leading models achieve a peak success rate of only ~66%, with a mean score around 55%. Models also exhibit domain-specific divergence, showing non-overlapping strengths in quantitative reasoning versus linguistic synthesis.. These findings underscore a significant "expert-gap" in current AI systems and establish XpertBench as a critical instrument for navigating the transition from general-purpose assistants to specialized professional collaborators. |
| title | Xpertbench: Expert Level Tasks with Rubrics-Based Evaluation |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2604.02368 |