Efficient MAP Estimation of LLM Judgment Performance with Prior Transfer
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915247594405888 |
|---|---|
| author | Qu, Huaizhi Choi, Inyoung Tan, Zhen Wang, Song Yun, Sukwon Long, Qi Siddiqui, Faizan Lee, Kwonjoon Chen, Tianlong |
| author_facet | Qu, Huaizhi Choi, Inyoung Tan, Zhen Wang, Song Yun, Sukwon Long, Qi Siddiqui, Faizan Lee, Kwonjoon Chen, Tianlong |
| contents | LLM ensembles are widely used for LLM judges. However, how to estimate their accuracy, especially in an efficient way, is unknown. In this paper, we present a principled maximum a posteriori (MAP) framework for an economical and precise estimation of the performance of LLM ensemble judgment. We first propose a mixture of Beta-Binomial distributions to model the judgment distribution, revising from the vanilla Binomial distribution. Next, we introduce a conformal prediction-driven approach that enables adaptive stopping during iterative sampling to balance accuracy with efficiency. Furthermore, we design a prior transfer mechanism that utilizes learned distributions on open-source datasets to improve estimation on a target dataset when only scarce annotations are available. Finally, we present BetaConform, a framework that integrates our distribution assumption, adaptive stopping, and the prior transfer mechanism to deliver a theoretically guaranteed distribution estimation of LLM ensemble judgment with minimum labeled samples. BetaConform is also validated empirically. For instance, with only 10 samples from the TruthfulQA dataset, for a Llama ensembled judge, BetaConform gauges its performance with error margin as small as 3.37%. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_12589 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Efficient MAP Estimation of LLM Judgment Performance with Prior Transfer Qu, Huaizhi Choi, Inyoung Tan, Zhen Wang, Song Yun, Sukwon Long, Qi Siddiqui, Faizan Lee, Kwonjoon Chen, Tianlong Machine Learning LLM ensembles are widely used for LLM judges. However, how to estimate their accuracy, especially in an efficient way, is unknown. In this paper, we present a principled maximum a posteriori (MAP) framework for an economical and precise estimation of the performance of LLM ensemble judgment. We first propose a mixture of Beta-Binomial distributions to model the judgment distribution, revising from the vanilla Binomial distribution. Next, we introduce a conformal prediction-driven approach that enables adaptive stopping during iterative sampling to balance accuracy with efficiency. Furthermore, we design a prior transfer mechanism that utilizes learned distributions on open-source datasets to improve estimation on a target dataset when only scarce annotations are available. Finally, we present BetaConform, a framework that integrates our distribution assumption, adaptive stopping, and the prior transfer mechanism to deliver a theoretically guaranteed distribution estimation of LLM ensemble judgment with minimum labeled samples. BetaConform is also validated empirically. For instance, with only 10 samples from the TruthfulQA dataset, for a Llama ensembled judge, BetaConform gauges its performance with error margin as small as 3.37%. |
| title | Efficient MAP Estimation of LLM Judgment Performance with Prior Transfer |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2504.12589 |