Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2504.00762 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915605233270784 |
|---|---|
| author | Chen, Jianhao Xun, Zishuo Zhou, Bocheng Qi, Han Zhang, Hangfan Zhang, Qiaosheng Chen, Yang Hu, Wei Qu, Yuzhong Ouyang, Wanli Hu, Shuyue |
| author_facet | Chen, Jianhao Xun, Zishuo Zhou, Bocheng Qi, Han Zhang, Hangfan Zhang, Qiaosheng Chen, Yang Hu, Wei Qu, Yuzhong Ouyang, Wanli Hu, Shuyue |
| contents | This paper presents a simple, effective, and cost-efficient strategy to improve LLM performance by scaling test-time compute. Our strategy builds upon the repeated-sampling-then-voting framework, with a novel twist: incorporating multiple models, even weaker ones, to leverage their complementary strengths that potentially arise from diverse training data and paradigms. By using consistency as a signal, our strategy dynamically switches between models. Theoretical analysis highlights the efficiency and performance advantages of our strategy. Extensive experiments on six datasets demonstrate that our strategy not only outperforms self-consistency and state-of-the-art multi-agent debate approaches, but also significantly reduces inference costs. Additionally, ModelSwitch requires only a few comparable LLMs to achieve optimal performance and can be extended with verification methods, demonstrating the potential of leveraging multiple LLMs in the generation-verification paradigm. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_00762 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Do We Truly Need So Many Samples? Multi-LLM Repeated Sampling Efficiently Scales Test-Time Compute Chen, Jianhao Xun, Zishuo Zhou, Bocheng Qi, Han Zhang, Hangfan Zhang, Qiaosheng Chen, Yang Hu, Wei Qu, Yuzhong Ouyang, Wanli Hu, Shuyue Artificial Intelligence This paper presents a simple, effective, and cost-efficient strategy to improve LLM performance by scaling test-time compute. Our strategy builds upon the repeated-sampling-then-voting framework, with a novel twist: incorporating multiple models, even weaker ones, to leverage their complementary strengths that potentially arise from diverse training data and paradigms. By using consistency as a signal, our strategy dynamically switches between models. Theoretical analysis highlights the efficiency and performance advantages of our strategy. Extensive experiments on six datasets demonstrate that our strategy not only outperforms self-consistency and state-of-the-art multi-agent debate approaches, but also significantly reduces inference costs. Additionally, ModelSwitch requires only a few comparable LLMs to achieve optimal performance and can be extended with verification methods, demonstrating the potential of leveraging multiple LLMs in the generation-verification paradigm. |
| title | Do We Truly Need So Many Samples? Multi-LLM Repeated Sampling Efficiently Scales Test-Time Compute |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2504.00762 |