Soup-of-Experts: Pretraining Specialist Models via Parameters Averaging
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866929696864731136 |
|---|---|
| author | Ablin, Pierre Katharopoulos, Angelos Seto, Skyler Grangier, David |
| author_facet | Ablin, Pierre Katharopoulos, Angelos Seto, Skyler Grangier, David |
| contents | Machine learning models are routinely trained on a mixture of different data domains. Different domain weights yield very different downstream performances. We propose the Soup-of-Experts, a novel architecture that can instantiate a model at test time for any domain weights with minimal computational cost and without re-training the model. Our architecture consists of a bank of expert parameters, which are linearly combined to instantiate one model. We learn the linear combination coefficients as a function of the input domain weights. To train this architecture, we sample random domain weights, instantiate the corresponding model, and backprop through one batch of data sampled with these domain weights. We demonstrate how our approach obtains small specialized models on several language modeling tasks quickly. Soup-of-Experts are particularly appealing when one needs to ship many different specialist models quickly under a model size constraint. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_01804 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Soup-of-Experts: Pretraining Specialist Models via Parameters Averaging Ablin, Pierre Katharopoulos, Angelos Seto, Skyler Grangier, David Machine Learning Computation and Language Machine learning models are routinely trained on a mixture of different data domains. Different domain weights yield very different downstream performances. We propose the Soup-of-Experts, a novel architecture that can instantiate a model at test time for any domain weights with minimal computational cost and without re-training the model. Our architecture consists of a bank of expert parameters, which are linearly combined to instantiate one model. We learn the linear combination coefficients as a function of the input domain weights. To train this architecture, we sample random domain weights, instantiate the corresponding model, and backprop through one batch of data sampled with these domain weights. We demonstrate how our approach obtains small specialized models on several language modeling tasks quickly. Soup-of-Experts are particularly appealing when one needs to ship many different specialist models quickly under a model size constraint. |
| title | Soup-of-Experts: Pretraining Specialist Models via Parameters Averaging |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2502.01804 |