RubricEval: A Rubric-Level Meta-Evaluation Benchmark for LLM Judges in Instruction Following
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910075176615936 |
|---|---|
| author | Pan, Tianjun Lin, Xuan Yang, Wenyan He, Qianyu Chen, Shisong Qi, Licai Xu, Wanqing Feng, Hongwei Xu, Bo Xiao, Yanghua |
| author_facet | Pan, Tianjun Lin, Xuan Yang, Wenyan He, Qianyu Chen, Shisong Qi, Licai Xu, Wanqing Feng, Hongwei Xu, Bo Xiao, Yanghua |
| contents | Rubric-based evaluation has become a prevailing paradigm for evaluating instruction following in large language models (LLMs). Despite its widespread use, the reliability of these rubric-level evaluations remains unclear, calling for meta-evaluation. However, prior meta-evaluation efforts largely focus on the response level, failing to assess the fine-grained judgment accuracy that rubric-based evaluation relies on. To bridge this gap, we introduce RubricEval. Our benchmark features: (1) the first rubric-level meta-evaluation benchmark for instruction following, (2) diverse instructions and responses spanning multiple categories and model sources, and (3) a substantial set of 3,486 quality-controlled instances, along with Easy/Hard subsets that better differentiates judge performance. Our experiments reveal that rubric-level judging remains far from solved: even GPT-4o, a widely adopted judge in instruction-following benchmarks, achieves only 55.97% on Hard subset. Considering evaluation paradigm, rubric-level evaluation outperforms checklist-level, explicit reasoning improves accuracy, and both together reduce inter-judge variance. Through our established rubric taxonomy, we further identify common failure modes and offer actionable insights for reliable instruction-following evaluation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_25133 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | RubricEval: A Rubric-Level Meta-Evaluation Benchmark for LLM Judges in Instruction Following Pan, Tianjun Lin, Xuan Yang, Wenyan He, Qianyu Chen, Shisong Qi, Licai Xu, Wanqing Feng, Hongwei Xu, Bo Xiao, Yanghua Artificial Intelligence Rubric-based evaluation has become a prevailing paradigm for evaluating instruction following in large language models (LLMs). Despite its widespread use, the reliability of these rubric-level evaluations remains unclear, calling for meta-evaluation. However, prior meta-evaluation efforts largely focus on the response level, failing to assess the fine-grained judgment accuracy that rubric-based evaluation relies on. To bridge this gap, we introduce RubricEval. Our benchmark features: (1) the first rubric-level meta-evaluation benchmark for instruction following, (2) diverse instructions and responses spanning multiple categories and model sources, and (3) a substantial set of 3,486 quality-controlled instances, along with Easy/Hard subsets that better differentiates judge performance. Our experiments reveal that rubric-level judging remains far from solved: even GPT-4o, a widely adopted judge in instruction-following benchmarks, achieves only 55.97% on Hard subset. Considering evaluation paradigm, rubric-level evaluation outperforms checklist-level, explicit reasoning improves accuracy, and both together reduce inter-judge variance. Through our established rubric taxonomy, we further identify common failure modes and offer actionable insights for reliable instruction-following evaluation. |
| title | RubricEval: A Rubric-Level Meta-Evaluation Benchmark for LLM Judges in Instruction Following |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2603.25133 |