Bench-CoE: a Framework for Collaboration of Experts from Benchmark
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866929616899276800 |
|---|---|
| author | Wang, Yuanshuai Zhang, Xingjian Zhao, Jinkun Wen, Siwei Feng, Peilin Liao, Shuhao Huang, Lei Wu, Wenjun |
| author_facet | Wang, Yuanshuai Zhang, Xingjian Zhao, Jinkun Wen, Siwei Feng, Peilin Liao, Shuhao Huang, Lei Wu, Wenjun |
| contents | Large Language Models (LLMs) are key technologies driving intelligent systems to handle multiple tasks. To meet the demands of various tasks, an increasing number of LLMs-driven experts with diverse capabilities have been developed, accompanied by corresponding benchmarks to evaluate their performance. This paper proposes the Bench-CoE framework, which enables Collaboration of Experts (CoE) by effectively leveraging benchmark evaluations to achieve optimal performance across various tasks. Bench-CoE includes a set of expert models, a router for assigning tasks to corresponding experts, and a benchmark dataset for training the router. Moreover, we formulate Query-Level and Subject-Level approaches based on our framework, and analyze the merits and drawbacks of these two approaches. Finally, we conduct a series of experiments with vary data distributions on both language and multimodal tasks to validate that our proposed Bench-CoE outperforms any single model in terms of overall performance. We hope this method serves as a baseline for further research in this area. The code is available at \url{https://github.com/ZhangXJ199/Bench-CoE}. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_04167 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Bench-CoE: a Framework for Collaboration of Experts from Benchmark Wang, Yuanshuai Zhang, Xingjian Zhao, Jinkun Wen, Siwei Feng, Peilin Liao, Shuhao Huang, Lei Wu, Wenjun Artificial Intelligence Large Language Models (LLMs) are key technologies driving intelligent systems to handle multiple tasks. To meet the demands of various tasks, an increasing number of LLMs-driven experts with diverse capabilities have been developed, accompanied by corresponding benchmarks to evaluate their performance. This paper proposes the Bench-CoE framework, which enables Collaboration of Experts (CoE) by effectively leveraging benchmark evaluations to achieve optimal performance across various tasks. Bench-CoE includes a set of expert models, a router for assigning tasks to corresponding experts, and a benchmark dataset for training the router. Moreover, we formulate Query-Level and Subject-Level approaches based on our framework, and analyze the merits and drawbacks of these two approaches. Finally, we conduct a series of experiments with vary data distributions on both language and multimodal tasks to validate that our proposed Bench-CoE outperforms any single model in terms of overall performance. We hope this method serves as a baseline for further research in this area. The code is available at \url{https://github.com/ZhangXJ199/Bench-CoE}. |
| title | Bench-CoE: a Framework for Collaboration of Experts from Benchmark |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2412.04167 |