Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2606.01629 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917553479090176 |
|---|---|
| author | Chen, Junjie Dong, Yuxi Li, Haitao Su, Weihang Zhou, Yujia Zhang, Min Liu, Yiqun Ai, Qinyao |
| author_facet | Chen, Junjie Dong, Yuxi Li, Haitao Su, Weihang Zhou, Yujia Zhang, Min Liu, Yiqun Ai, Qinyao |
| contents | As large language models (LLMs) are increasingly used for long-form generation, reliably evaluating long-form outputs has become a critical challenge. LLM-as-a-judge offers a scalable alternative to human evaluation, yet its reliability in long-form output evaluation remains underexamined: existing meta-evaluation benchmarks focus mainly on short-form outputs. Compared with short-form evaluation, long-form evaluation is not merely a matter of output length; it often requires judges to handle more complex document-level demands. In this work, we introduce LongJudgeBench, a comprehensive benchmark for evaluating LLM judges on long-form outputs across diverse real-world scenarios and judging protocols. We systematically evaluate a broad range of LLM judges, covering multiple base models and judging settings. Our results reveal a substantial reliability gap: current LLM judges remain unstable across scenarios, and rubrics or references are helpful but not always sufficient. We hope LongJudgeBench will support future research on more robust, context-aware, and human-aligned LLM-as-a-judge methods. Our code is available at https://anonymous.4open.science/r/LongJudgeBench-F782. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2606_01629 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Benchmarking LLM-as-a-Judge for Long-Form Output Evaluation Chen, Junjie Dong, Yuxi Li, Haitao Su, Weihang Zhou, Yujia Zhang, Min Liu, Yiqun Ai, Qinyao Computation and Language As large language models (LLMs) are increasingly used for long-form generation, reliably evaluating long-form outputs has become a critical challenge. LLM-as-a-judge offers a scalable alternative to human evaluation, yet its reliability in long-form output evaluation remains underexamined: existing meta-evaluation benchmarks focus mainly on short-form outputs. Compared with short-form evaluation, long-form evaluation is not merely a matter of output length; it often requires judges to handle more complex document-level demands. In this work, we introduce LongJudgeBench, a comprehensive benchmark for evaluating LLM judges on long-form outputs across diverse real-world scenarios and judging protocols. We systematically evaluate a broad range of LLM judges, covering multiple base models and judging settings. Our results reveal a substantial reliability gap: current LLM judges remain unstable across scenarios, and rubrics or references are helpful but not always sufficient. We hope LongJudgeBench will support future research on more robust, context-aware, and human-aligned LLM-as-a-judge methods. Our code is available at https://anonymous.4open.science/r/LongJudgeBench-F782. |
| title | Benchmarking LLM-as-a-Judge for Long-Form Output Evaluation |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2606.01629 |