Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2403.16854 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909221501534208 |
|---|---|
| author | Chai, Ziwei Wang, Guoyin Su, Jing Zhang, Tianjie Huang, Xuanwen Wang, Xuwu Xu, Jingjing Yuan, Jianbo Yang, Hongxia Wu, Fei Yang, Yang |
| author_facet | Chai, Ziwei Wang, Guoyin Su, Jing Zhang, Tianjie Huang, Xuanwen Wang, Xuwu Xu, Jingjing Yuan, Jianbo Yang, Hongxia Wu, Fei Yang, Yang |
| contents | We present Expert-Token-Routing, a unified generalist framework that facilitates seamless integration of multiple expert LLMs. Our framework represents expert LLMs as special expert tokens within the vocabulary of a meta LLM. The meta LLM can route to an expert LLM like generating new tokens. Expert-Token-Routing not only supports learning the implicit expertise of expert LLMs from existing instruction dataset but also allows for dynamic extension of new expert LLMs in a plug-and-play manner. It also conceals the detailed collaboration process from the user's perspective, facilitating interaction as though it were a singular LLM. Our framework outperforms various existing multi-LLM collaboration paradigms across benchmarks that incorporate six diverse expert domains, demonstrating effectiveness and robustness in building generalist LLM system via synergizing multiple expert LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_16854 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | An Expert is Worth One Token: Synergizing Multiple Expert LLMs as Generalist via Expert Token Routing Chai, Ziwei Wang, Guoyin Su, Jing Zhang, Tianjie Huang, Xuanwen Wang, Xuwu Xu, Jingjing Yuan, Jianbo Yang, Hongxia Wu, Fei Yang, Yang Computation and Language Artificial Intelligence We present Expert-Token-Routing, a unified generalist framework that facilitates seamless integration of multiple expert LLMs. Our framework represents expert LLMs as special expert tokens within the vocabulary of a meta LLM. The meta LLM can route to an expert LLM like generating new tokens. Expert-Token-Routing not only supports learning the implicit expertise of expert LLMs from existing instruction dataset but also allows for dynamic extension of new expert LLMs in a plug-and-play manner. It also conceals the detailed collaboration process from the user's perspective, facilitating interaction as though it were a singular LLM. Our framework outperforms various existing multi-LLM collaboration paradigms across benchmarks that incorporate six diverse expert domains, demonstrating effectiveness and robustness in building generalist LLM system via synergizing multiple expert LLMs. |
| title | An Expert is Worth One Token: Synergizing Multiple Expert LLMs as Generalist via Expert Token Routing |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2403.16854 |