RubricBench: Aligning Model-Generated Rubrics with Human Standards
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910038754328576 |
|---|---|
| author | Zhang, Qiyuan Zhou, Junyi Wang, Yufei Lyu, Fuyuan Ming, Yidong Xu, Can Sun, Qingfeng Zheng, Kai Kang, Peng Liu, Xue Ma, Chen |
| author_facet | Zhang, Qiyuan Zhou, Junyi Wang, Yufei Lyu, Fuyuan Ming, Yidong Xu, Can Sun, Qingfeng Zheng, Kai Kang, Peng Liu, Xue Ma, Chen |
| contents | As Large Language Model (LLM) alignment evolves from simple completions to complex, highly sophisticated generation, Reward Models are increasingly shifting toward rubric-guided evaluation to mitigate surface-level biases. However, the community lacks a unified benchmark to assess this evaluation paradigm, as existing benchmarks lack both the discriminative complexity and the ground-truth rubric annotations required for rigorous analysis. To bridge this gap, we introduce RubricBench, a curated benchmark with 1,147 pairwise comparisons specifically designed to assess the reliability of rubric-based evaluation. Our construction employs a multi-dimensional filtration pipeline to target hard samples featuring nuanced input complexity and misleading surface bias, augmenting each with expert-annotated, atomic rubrics derived strictly from instructions. Comprehensive experiments reveal a substantial capability gap between human-annotated and model-generated rubrics, indicating that even state-of-the-art models struggle to autonomously specify valid evaluation criteria, lagging considerably behind human-guided performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_01562 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | RubricBench: Aligning Model-Generated Rubrics with Human Standards Zhang, Qiyuan Zhou, Junyi Wang, Yufei Lyu, Fuyuan Ming, Yidong Xu, Can Sun, Qingfeng Zheng, Kai Kang, Peng Liu, Xue Ma, Chen Artificial Intelligence As Large Language Model (LLM) alignment evolves from simple completions to complex, highly sophisticated generation, Reward Models are increasingly shifting toward rubric-guided evaluation to mitigate surface-level biases. However, the community lacks a unified benchmark to assess this evaluation paradigm, as existing benchmarks lack both the discriminative complexity and the ground-truth rubric annotations required for rigorous analysis. To bridge this gap, we introduce RubricBench, a curated benchmark with 1,147 pairwise comparisons specifically designed to assess the reliability of rubric-based evaluation. Our construction employs a multi-dimensional filtration pipeline to target hard samples featuring nuanced input complexity and misleading surface bias, augmenting each with expert-annotated, atomic rubrics derived strictly from instructions. Comprehensive experiments reveal a substantial capability gap between human-annotated and model-generated rubrics, indicating that even state-of-the-art models struggle to autonomously specify valid evaluation criteria, lagging considerably behind human-guided performance. |
| title | RubricBench: Aligning Model-Generated Rubrics with Human Standards |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2603.01562 |