Crowd Comparative Reasoning: Unlocking Comprehensive Evaluations for LLM-as-a-Judge
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910904947310592 |
|---|---|
| author | Zhang, Qiyuan Wang, Yufei Jiang, Yuxin Li, Liangyou Wu, Chuhan Wang, Yasheng Jiang, Xin Shang, Lifeng Tang, Ruiming Lyu, Fuyuan Ma, Chen |
| author_facet | Zhang, Qiyuan Wang, Yufei Jiang, Yuxin Li, Liangyou Wu, Chuhan Wang, Yasheng Jiang, Xin Shang, Lifeng Tang, Ruiming Lyu, Fuyuan Ma, Chen |
| contents | LLM-as-a-Judge, which generates chain-of-thought (CoT) judgments, has become a widely adopted auto-evaluation method. However, its reliability is compromised by the CoT reasoning's inability to capture comprehensive and deeper details, often leading to incomplete outcomes. Existing methods mainly rely on majority voting or criteria expansion, which is insufficient to address the limitation in CoT. We propose Crowd-based Comparative Evaluation, which introduces additional crowd responses to compare with the candidate responses, thereby exposing deeper and more comprehensive details within the candidate responses. This process effectively guides LLM-as-a-Judge to provide a more detailed CoT judgment. Extensive experiments demonstrate that our approach enhances evaluation reliability, achieving an average accuracy gain of 6.7% across five benchmarks. Moreover, our method produces higher-quality CoTs that facilitate judge distillation and exhibit superior performance in rejection sampling for supervised fine-tuning (SFT), referred to as crowd rejection sampling, thereby enabling more efficient SFT. Our analysis confirms that CoTs generated by ours are more comprehensive and of higher quality, and evaluation accuracy improves as inference scales. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_12501 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Crowd Comparative Reasoning: Unlocking Comprehensive Evaluations for LLM-as-a-Judge Zhang, Qiyuan Wang, Yufei Jiang, Yuxin Li, Liangyou Wu, Chuhan Wang, Yasheng Jiang, Xin Shang, Lifeng Tang, Ruiming Lyu, Fuyuan Ma, Chen Computation and Language LLM-as-a-Judge, which generates chain-of-thought (CoT) judgments, has become a widely adopted auto-evaluation method. However, its reliability is compromised by the CoT reasoning's inability to capture comprehensive and deeper details, often leading to incomplete outcomes. Existing methods mainly rely on majority voting or criteria expansion, which is insufficient to address the limitation in CoT. We propose Crowd-based Comparative Evaluation, which introduces additional crowd responses to compare with the candidate responses, thereby exposing deeper and more comprehensive details within the candidate responses. This process effectively guides LLM-as-a-Judge to provide a more detailed CoT judgment. Extensive experiments demonstrate that our approach enhances evaluation reliability, achieving an average accuracy gain of 6.7% across five benchmarks. Moreover, our method produces higher-quality CoTs that facilitate judge distillation and exhibit superior performance in rejection sampling for supervised fine-tuning (SFT), referred to as crowd rejection sampling, thereby enabling more efficient SFT. Our analysis confirms that CoTs generated by ours are more comprehensive and of higher quality, and evaluation accuracy improves as inference scales. |
| title | Crowd Comparative Reasoning: Unlocking Comprehensive Evaluations for LLM-as-a-Judge |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2502.12501 |