Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2602.08003 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910015616450560 |
|---|---|
| author | Turkmen, Yigit Buyukates, Baturalp Bastopcu, Melih |
| author_facet | Turkmen, Yigit Buyukates, Baturalp Bastopcu, Melih |
| contents | Large language models (LLMs) are often ensembled together to improve overall reliability and robustness, but in practice models are strongly correlated. This raises a fundamental question: which models should be selected when forming an LLM ensemble? We formulate budgeted ensemble selection as maximizing the mutual information between the true label and predictions of the selected models. Furthermore, to explain why performance can saturate even with many models, we model the correlated errors of the models using Gaussian-copula and show an information-theoretic error floor for the performance of the ensemble. Motivated by these, we propose a simple greedy mutual-information selection algorithm that estimates the required information terms directly from data and iteratively builds an ensemble under a query budget. We test our approach in two question answering datasets and one binary sentiment classification dataset: MEDMCQA, MMLU, and IMDB movie reviews. Across all datasets, we observe that our method consistently outperforms strong baselines under the same query budget. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_08003 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Don't Always Pick the Highest-Performing Model: An Information Theoretic View of LLM Ensemble Selection Turkmen, Yigit Buyukates, Baturalp Bastopcu, Melih Machine Learning Artificial Intelligence Distributed, Parallel, and Cluster Computing Information Theory Large language models (LLMs) are often ensembled together to improve overall reliability and robustness, but in practice models are strongly correlated. This raises a fundamental question: which models should be selected when forming an LLM ensemble? We formulate budgeted ensemble selection as maximizing the mutual information between the true label and predictions of the selected models. Furthermore, to explain why performance can saturate even with many models, we model the correlated errors of the models using Gaussian-copula and show an information-theoretic error floor for the performance of the ensemble. Motivated by these, we propose a simple greedy mutual-information selection algorithm that estimates the required information terms directly from data and iteratively builds an ensemble under a query budget. We test our approach in two question answering datasets and one binary sentiment classification dataset: MEDMCQA, MMLU, and IMDB movie reviews. Across all datasets, we observe that our method consistently outperforms strong baselines under the same query budget. |
| title | Don't Always Pick the Highest-Performing Model: An Information Theoretic View of LLM Ensemble Selection |
| topic | Machine Learning Artificial Intelligence Distributed, Parallel, and Cluster Computing Information Theory |
| url | https://arxiv.org/abs/2602.08003 |