Scoring, Reasoning, and Selecting the Best! Ensembling Large Language Models via a Peer-Review Process
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917435805794304 |
|---|---|
| author | Chen, Zhijun Ji, Zeyu Mao, Qianren Wu, Hao Song, Jinhuan Cheng, Junhang Qin, Bangjie Li, Zhuoran Li, Jingzheng Sun, Kai Wang, Zizhe Ban, Yikun Sun, Zhu Ji, Xiangyang Sun, Hailong |
| author_facet | Chen, Zhijun Ji, Zeyu Mao, Qianren Wu, Hao Song, Jinhuan Cheng, Junhang Qin, Bangjie Li, Zhuoran Li, Jingzheng Sun, Kai Wang, Zizhe Ban, Yikun Sun, Zhu Ji, Xiangyang Sun, Hailong |
| contents | We propose LLM-PeerReview, an unsupervised LLM Ensemble method that selects the most ideal response from multiple LLM-generated candidates for each query, harnessing the collective wisdom of multiple models with diverse strengths. LLM-PeerReview is built on a novel, peer-review-inspired framework that offers a transparent and interpretable mechanism, while remaining fully unsupervised for flexible adaptability and generalization. Specifically, it operates in three stages: For scoring, we use the emerging LLM-as-a-Judge technique to evaluate each response by reusing multiple LLMs at hand; For reasoning, we can apply a straightforward averaging strategy or a principled graphical model-based truth inference algorithm to aggregate multiple scores to produce a final score for each response; Finally, the highest-scoring response is selected as the best ensemble output. LLM-PeerReview is conceptually simple and empirically powerful. Our results across four datasets show that the two variants of the proposed approach outperform the advanced model Smoothie-Global by 6.9% and 7.3% points, cross diverse task types including factual recall QA, math reasoning, and instruction following. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_23213 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Scoring, Reasoning, and Selecting the Best! Ensembling Large Language Models via a Peer-Review Process Chen, Zhijun Ji, Zeyu Mao, Qianren Wu, Hao Song, Jinhuan Cheng, Junhang Qin, Bangjie Li, Zhuoran Li, Jingzheng Sun, Kai Wang, Zizhe Ban, Yikun Sun, Zhu Ji, Xiangyang Sun, Hailong Computation and Language Artificial Intelligence We propose LLM-PeerReview, an unsupervised LLM Ensemble method that selects the most ideal response from multiple LLM-generated candidates for each query, harnessing the collective wisdom of multiple models with diverse strengths. LLM-PeerReview is built on a novel, peer-review-inspired framework that offers a transparent and interpretable mechanism, while remaining fully unsupervised for flexible adaptability and generalization. Specifically, it operates in three stages: For scoring, we use the emerging LLM-as-a-Judge technique to evaluate each response by reusing multiple LLMs at hand; For reasoning, we can apply a straightforward averaging strategy or a principled graphical model-based truth inference algorithm to aggregate multiple scores to produce a final score for each response; Finally, the highest-scoring response is selected as the best ensemble output. LLM-PeerReview is conceptually simple and empirically powerful. Our results across four datasets show that the two variants of the proposed approach outperform the advanced model Smoothie-Global by 6.9% and 7.3% points, cross diverse task types including factual recall QA, math reasoning, and instruction following. |
| title | Scoring, Reasoning, and Selecting the Best! Ensembling Large Language Models via a Peer-Review Process |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2512.23213 |