Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2603.00993 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910036868988928 |
|---|---|
| author | Qian, Yiyue Zhang, Shinan Zhou, Yun Ding, Haibo Socolinsky, Diego Zhang, Yi |
| author_facet | Qian, Yiyue Zhang, Shinan Zhou, Yun Ding, Haibo Socolinsky, Diego Zhang, Yi |
| contents | Large Language Models (LLMs) have revolutionized AI-generated content evaluation, with the LLM-as-a-Judge paradigm becoming increasingly popular. However, current single-LLM evaluation approaches face significant challenges, including inconsistent judgments and inherent biases from pre-training data. To address these limitations, we propose CollabEval, a novel multi-agent evaluation framework that implements a three-phase Collaborative Evaluation process: initial evaluation, multi-round discussion, and final judgment. Unlike existing approaches that rely on competitive debate or single-model evaluation, CollabEval emphasizes collaboration among multiple agents with strategic consensus checking for efficiency. Our extensive experiments demonstrate that CollabEval consistently outperforms single-LLM approaches across multiple dimensions while maintaining robust performance even when individual models struggle. The framework provides comprehensive support for various evaluation criteria while ensuring efficiency through its collaborative design. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_00993 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | CollabEval: Enhancing LLM-as-a-Judge via Multi-Agent Collaboration Qian, Yiyue Zhang, Shinan Zhou, Yun Ding, Haibo Socolinsky, Diego Zhang, Yi Artificial Intelligence Large Language Models (LLMs) have revolutionized AI-generated content evaluation, with the LLM-as-a-Judge paradigm becoming increasingly popular. However, current single-LLM evaluation approaches face significant challenges, including inconsistent judgments and inherent biases from pre-training data. To address these limitations, we propose CollabEval, a novel multi-agent evaluation framework that implements a three-phase Collaborative Evaluation process: initial evaluation, multi-round discussion, and final judgment. Unlike existing approaches that rely on competitive debate or single-model evaluation, CollabEval emphasizes collaboration among multiple agents with strategic consensus checking for efficiency. Our extensive experiments demonstrate that CollabEval consistently outperforms single-LLM approaches across multiple dimensions while maintaining robust performance even when individual models struggle. The framework provides comprehensive support for various evaluation criteria while ensuring efficiency through its collaborative design. |
| title | CollabEval: Enhancing LLM-as-a-Judge via Multi-Agent Collaboration |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2603.00993 |