Determine-Then-Ensemble: Necessity of Top-k Union for Large Language Model Ensembling
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909510095863808 |
|---|---|
| author | Yao, Yuxuan Wu, Han Liu, Mingyang Luo, Sichun Han, Xiongwei Liu, Jie Guo, Zhijiang Song, Linqi |
| author_facet | Yao, Yuxuan Wu, Han Liu, Mingyang Luo, Sichun Han, Xiongwei Liu, Jie Guo, Zhijiang Song, Linqi |
| contents | Large language models (LLMs) exhibit varying strengths and weaknesses across different tasks, prompting recent studies to explore the benefits of ensembling models to leverage their complementary advantages. However, existing LLM ensembling methods often overlook model compatibility and struggle with inefficient alignment of probabilities across the entire vocabulary. In this study, we empirically investigate the factors influencing ensemble performance, identifying model performance, vocabulary size, and response style as key determinants, revealing that compatibility among models is essential for effective ensembling. This analysis leads to the development of a simple yet effective model selection strategy that identifies compatible models. Additionally, we introduce the \textsc{Uni}on \textsc{T}op-$k$ \textsc{E}nsembling (\textsc{UniTE}), a novel approach that efficiently combines models by focusing on the union of the top-k tokens from each model, thereby avoiding the need for full vocabulary alignment and reducing computational overhead. Extensive evaluations across multiple benchmarks demonstrate that \textsc{UniTE} significantly enhances performance compared to existing methods, offering a more efficient framework for LLM ensembling. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_03777 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Determine-Then-Ensemble: Necessity of Top-k Union for Large Language Model Ensembling Yao, Yuxuan Wu, Han Liu, Mingyang Luo, Sichun Han, Xiongwei Liu, Jie Guo, Zhijiang Song, Linqi Computation and Language Artificial Intelligence Large language models (LLMs) exhibit varying strengths and weaknesses across different tasks, prompting recent studies to explore the benefits of ensembling models to leverage their complementary advantages. However, existing LLM ensembling methods often overlook model compatibility and struggle with inefficient alignment of probabilities across the entire vocabulary. In this study, we empirically investigate the factors influencing ensemble performance, identifying model performance, vocabulary size, and response style as key determinants, revealing that compatibility among models is essential for effective ensembling. This analysis leads to the development of a simple yet effective model selection strategy that identifies compatible models. Additionally, we introduce the \textsc{Uni}on \textsc{T}op-$k$ \textsc{E}nsembling (\textsc{UniTE}), a novel approach that efficiently combines models by focusing on the union of the top-k tokens from each model, thereby avoiding the need for full vocabulary alignment and reducing computational overhead. Extensive evaluations across multiple benchmarks demonstrate that \textsc{UniTE} significantly enhances performance compared to existing methods, offering a more efficient framework for LLM ensembling. |
| title | Determine-Then-Ensemble: Necessity of Top-k Union for Large Language Model Ensembling |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2410.03777 |