Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2502.08788 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913906231869440 |
|---|---|
| author | Zhang, Hangfan Cui, Zhiyao Chen, Jianhao Wang, Xinrun Zhang, Qiaosheng Wang, Zhen Wu, Dinghao Hu, Shuyue |
| author_facet | Zhang, Hangfan Cui, Zhiyao Chen, Jianhao Wang, Xinrun Zhang, Qiaosheng Wang, Zhen Wu, Dinghao Hu, Shuyue |
| contents | Multi-agent debate (MAD) has gained significant attention as a promising line of research to improve the factual accuracy and reasoning capabilities of large language models (LLMs). Despite its conceptual appeal, current MAD research suffers from critical limitations in evaluation practices, including limited benchmark coverage, weak baseline comparisons, and inconsistent setups. This paper presents a systematic evaluation of 5 representative MAD methods across 9 benchmarks using 4 foundational models. Surprisingly, our findings reveal that MAD often fail to outperform simple single-agent baselines such as Chain-of-Thought and Self-Consistency, even when consuming significantly more inference-time computation. To advance MAD research, we further explore the role of model heterogeneity and find it as a universal antidote to consistently improve current MAD frameworks. Based on our findings, we argue that the field must stop overvaluing MAD in its current form; for true advancement, we must critically rethink evaluation paradigms and actively embrace model heterogeneity as a core design principle. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_08788 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Stop Overvaluing Multi-Agent Debate -- We Must Rethink Evaluation and Embrace Model Heterogeneity Zhang, Hangfan Cui, Zhiyao Chen, Jianhao Wang, Xinrun Zhang, Qiaosheng Wang, Zhen Wu, Dinghao Hu, Shuyue Computation and Language Machine Learning Multi-agent debate (MAD) has gained significant attention as a promising line of research to improve the factual accuracy and reasoning capabilities of large language models (LLMs). Despite its conceptual appeal, current MAD research suffers from critical limitations in evaluation practices, including limited benchmark coverage, weak baseline comparisons, and inconsistent setups. This paper presents a systematic evaluation of 5 representative MAD methods across 9 benchmarks using 4 foundational models. Surprisingly, our findings reveal that MAD often fail to outperform simple single-agent baselines such as Chain-of-Thought and Self-Consistency, even when consuming significantly more inference-time computation. To advance MAD research, we further explore the role of model heterogeneity and find it as a universal antidote to consistently improve current MAD frameworks. Based on our findings, we argue that the field must stop overvaluing MAD in its current form; for true advancement, we must critically rethink evaluation paradigms and actively embrace model heterogeneity as a core design principle. |
| title | Stop Overvaluing Multi-Agent Debate -- We Must Rethink Evaluation and Embrace Model Heterogeneity |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2502.08788 |