Synergistic Weak-Strong Collaboration by Aligning Preferences
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916701322346496 |
|---|---|
| author | Jiao, Yizhu Zhang, Xuchao Wang, Zhaoyang Ma, Yubo Deng, Zhun Wang, Rujia Bansal, Chetan Rajmohan, Saravan Han, Jiawei Yao, Huaxiu |
| author_facet | Jiao, Yizhu Zhang, Xuchao Wang, Zhaoyang Ma, Yubo Deng, Zhun Wang, Rujia Bansal, Chetan Rajmohan, Saravan Han, Jiawei Yao, Huaxiu |
| contents | Current Large Language Models (LLMs) excel in general reasoning yet struggle with specialized tasks requiring proprietary or domain-specific knowledge. Fine-tuning large models for every niche application is often infeasible due to black-box constraints and high computational overhead. To address this, we propose a collaborative framework that pairs a specialized weak model with a general strong model. The weak model, tailored to specific domains, produces initial drafts and background information, while the strong model leverages its advanced reasoning to refine these drafts, extending LLMs' capabilities to critical yet specialized tasks. To optimize this collaboration, we introduce a collaborative feedback to fine-tunes the weak model, which quantifies the influence of the weak model's contributions in the collaboration procedure and establishes preference pairs to guide preference tuning of the weak model. We validate our framework through experiments on three domains. We find that the collaboration significantly outperforms each model alone by leveraging complementary strengths. Moreover, aligning the weak model with the collaborative preference further enhances overall performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_15188 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Synergistic Weak-Strong Collaboration by Aligning Preferences Jiao, Yizhu Zhang, Xuchao Wang, Zhaoyang Ma, Yubo Deng, Zhun Wang, Rujia Bansal, Chetan Rajmohan, Saravan Han, Jiawei Yao, Huaxiu Artificial Intelligence Current Large Language Models (LLMs) excel in general reasoning yet struggle with specialized tasks requiring proprietary or domain-specific knowledge. Fine-tuning large models for every niche application is often infeasible due to black-box constraints and high computational overhead. To address this, we propose a collaborative framework that pairs a specialized weak model with a general strong model. The weak model, tailored to specific domains, produces initial drafts and background information, while the strong model leverages its advanced reasoning to refine these drafts, extending LLMs' capabilities to critical yet specialized tasks. To optimize this collaboration, we introduce a collaborative feedback to fine-tunes the weak model, which quantifies the influence of the weak model's contributions in the collaboration procedure and establishes preference pairs to guide preference tuning of the weak model. We validate our framework through experiments on three domains. We find that the collaboration significantly outperforms each model alone by leveraging complementary strengths. Moreover, aligning the weak model with the collaborative preference further enhances overall performance. |
| title | Synergistic Weak-Strong Collaboration by Aligning Preferences |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2504.15188 |