Saved in:
| Main Authors: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2509.23537 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912622166671360 |
|---|---|
| author | Tian, Aaron Xuxiang Zhang, Ruofan Tang, Jiayao Cho, Young Min Li, Xueqian Yi, Qiang Wang, Ji Zhang, Zhunping Qi, Danrui Li, Zekun Xiang, Xingyu Guntuku, Sharath Chandra Ungar, Lyle Shi, Tianyu Wang, Chi |
| author_facet | Tian, Aaron Xuxiang Zhang, Ruofan Tang, Jiayao Cho, Young Min Li, Xueqian Yi, Qiang Wang, Ji Zhang, Zhunping Qi, Danrui Li, Zekun Xiang, Xingyu Guntuku, Sharath Chandra Ungar, Lyle Shi, Tianyu Wang, Chi |
| contents | We study multi-turn multi-agent orchestration, where multiple large language model (LLM) agents interact over multiple turns by iteratively proposing answers or casting votes until reaching consensus. Using four LLMs (Gemini 2.5 Pro, GPT-5, Grok 4, and Claude Sonnet 4) on GPQA-Diamond, IFEval, and MuSR, we conduct two experiments: (i) benchmarking orchestration against single-LLM baselines; and (ii) ablations on GPQA-Diamond that vary whether agents see who authored answers and whether they can observe ongoing votes. Orchestration matches or exceeds the strongest single model and consistently outperforms the others. Analysis of best-achievable orchestration performance shows potential for further gains. The ablations show that revealing authorship increases self-voting and ties, and that showing ongoing votes amplifies herding, which speeds convergence but can sometimes yield premature consensus. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_23537 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Beyond the Strongest LLM: Multi-Turn Multi-Agent Orchestration vs. Single LLMs on Benchmarks Tian, Aaron Xuxiang Zhang, Ruofan Tang, Jiayao Cho, Young Min Li, Xueqian Yi, Qiang Wang, Ji Zhang, Zhunping Qi, Danrui Li, Zekun Xiang, Xingyu Guntuku, Sharath Chandra Ungar, Lyle Shi, Tianyu Wang, Chi Artificial Intelligence We study multi-turn multi-agent orchestration, where multiple large language model (LLM) agents interact over multiple turns by iteratively proposing answers or casting votes until reaching consensus. Using four LLMs (Gemini 2.5 Pro, GPT-5, Grok 4, and Claude Sonnet 4) on GPQA-Diamond, IFEval, and MuSR, we conduct two experiments: (i) benchmarking orchestration against single-LLM baselines; and (ii) ablations on GPQA-Diamond that vary whether agents see who authored answers and whether they can observe ongoing votes. Orchestration matches or exceeds the strongest single model and consistently outperforms the others. Analysis of best-achievable orchestration performance shows potential for further gains. The ablations show that revealing authorship increases self-voting and ties, and that showing ongoing votes amplifies herding, which speeds convergence but can sometimes yield premature consensus. |
| title | Beyond the Strongest LLM: Multi-Turn Multi-Agent Orchestration vs. Single LLMs on Benchmarks |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2509.23537 |