Flexible and Effective Mixing of Large Language Models into a Mixture of Domain Experts
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909311203016704 |
|---|---|
| author | Lee, Rhui Dih Wynter, Laura Ganti, Raghu Kiran |
| author_facet | Lee, Rhui Dih Wynter, Laura Ganti, Raghu Kiran |
| contents | We present a toolkit for creating low-cost Mixture-of-Domain-Experts (MOE) from trained models. The toolkit can be used for creating a mixture from models or from adapters. We perform extensive tests and offer guidance on defining the architecture of the resulting MOE using the toolkit. A public repository is available. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_17280 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Flexible and Effective Mixing of Large Language Models into a Mixture of Domain Experts Lee, Rhui Dih Wynter, Laura Ganti, Raghu Kiran Artificial Intelligence Computation and Language We present a toolkit for creating low-cost Mixture-of-Domain-Experts (MOE) from trained models. The toolkit can be used for creating a mixture from models or from adapters. We perform extensive tests and offer guidance on defining the architecture of the resulting MOE using the toolkit. A public repository is available. |
| title | Flexible and Effective Mixing of Large Language Models into a Mixture of Domain Experts |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2408.17280 |