Heterogeneous Swarms: Jointly Optimizing Model Roles and Weights for Multi-LLM Systems
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866918165888368640 |
|---|---|
| author | Feng, Shangbin Wang, Zifeng Goyal, Palash Wang, Yike Shi, Weijia Xia, Huang Palangi, Hamid Zettlemoyer, Luke Tsvetkov, Yulia Lee, Chen-Yu Pfister, Tomas |
| author_facet | Feng, Shangbin Wang, Zifeng Goyal, Palash Wang, Yike Shi, Weijia Xia, Huang Palangi, Hamid Zettlemoyer, Luke Tsvetkov, Yulia Lee, Chen-Yu Pfister, Tomas |
| contents | We propose Heterogeneous Swarms, an algorithm to design multi-LLM systems by jointly optimizing model roles and weights. We represent multi-LLM systems as directed acyclic graphs (DAGs) of LLMs with topological message passing for collaborative generation. Given a pool of LLM experts and a utility function, Heterogeneous Swarms employs two iterative steps: role-step and weight-step. For role-step, we interpret model roles as learning a DAG that specifies the flow of inputs and outputs between LLMs. Starting from a swarm of random continuous adjacency matrices, we decode them into discrete DAGs, call the LLMs in topological order, evaluate on the utility function (e.g. accuracy on a task), and optimize the adjacency matrices with particle swarm optimization based on the utility score. For weight-step, we assess the contribution of individual LLMs in the multi-LLM systems and optimize model weights with swarm intelligence. We propose JFK-score to quantify the individual contribution of each LLM in the best-found DAG of the role-step, then optimize model weights with particle swarm optimization based on the JFK-score. Experiments demonstrate that Heterogeneous Swarms outperforms 15 role- and/or weight-based baselines by 18.5% on average across 12 tasks. Further analysis reveals that Heterogeneous Swarms discovers multi-LLM systems with heterogeneous model roles and substantial collaborative gains, and benefits from the diversity of language models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_04510 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Heterogeneous Swarms: Jointly Optimizing Model Roles and Weights for Multi-LLM Systems Feng, Shangbin Wang, Zifeng Goyal, Palash Wang, Yike Shi, Weijia Xia, Huang Palangi, Hamid Zettlemoyer, Luke Tsvetkov, Yulia Lee, Chen-Yu Pfister, Tomas Computation and Language We propose Heterogeneous Swarms, an algorithm to design multi-LLM systems by jointly optimizing model roles and weights. We represent multi-LLM systems as directed acyclic graphs (DAGs) of LLMs with topological message passing for collaborative generation. Given a pool of LLM experts and a utility function, Heterogeneous Swarms employs two iterative steps: role-step and weight-step. For role-step, we interpret model roles as learning a DAG that specifies the flow of inputs and outputs between LLMs. Starting from a swarm of random continuous adjacency matrices, we decode them into discrete DAGs, call the LLMs in topological order, evaluate on the utility function (e.g. accuracy on a task), and optimize the adjacency matrices with particle swarm optimization based on the utility score. For weight-step, we assess the contribution of individual LLMs in the multi-LLM systems and optimize model weights with swarm intelligence. We propose JFK-score to quantify the individual contribution of each LLM in the best-found DAG of the role-step, then optimize model weights with particle swarm optimization based on the JFK-score. Experiments demonstrate that Heterogeneous Swarms outperforms 15 role- and/or weight-based baselines by 18.5% on average across 12 tasks. Further analysis reveals that Heterogeneous Swarms discovers multi-LLM systems with heterogeneous model roles and substantial collaborative gains, and benefits from the diversity of language models. |
| title | Heterogeneous Swarms: Jointly Optimizing Model Roles and Weights for Multi-LLM Systems |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2502.04510 |