UniCBE: An Uniformity-driven Comparing Based Evaluation Framework with Unified Multi-Objective Optimization
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912233708060672 |
|---|---|
| author | Yuan, Peiwen Feng, Shaoxiong Li, Yiwei Wang, Xinglin Zhang, Yueqi Shi, Jiayi Tan, Chuyi Pan, Boyuan Hu, Yao Li, Kan |
| author_facet | Yuan, Peiwen Feng, Shaoxiong Li, Yiwei Wang, Xinglin Zhang, Yueqi Shi, Jiayi Tan, Chuyi Pan, Boyuan Hu, Yao Li, Kan |
| contents | Human preference plays a significant role in measuring large language models and guiding them to align with human values. Unfortunately, current comparing-based evaluation (CBE) methods typically focus on a single optimization objective, failing to effectively utilize scarce yet valuable preference signals. To address this, we delve into key factors that can enhance the accuracy, convergence, and scalability of CBE: suppressing sampling bias, balancing descending process of uncertainty, and mitigating updating uncertainty. Following the derived guidelines, we propose UniCBE, a unified uniformity-driven CBE framework which simultaneously optimize these core objectives by constructing and integrating three decoupled sampling probability matrices, each designed to ensure uniformity in specific aspects. We further ablate the optimal tuple sampling and preference aggregation strategies to achieve efficient CBE. On the AlpacaEval benchmark, UniCBE saves over 17% of evaluation budgets while achieving a Pearson correlation with ground truth exceeding 0.995, demonstrating excellent accuracy and convergence. In scenarios where new models are continuously introduced, UniCBE can even save over 50% of evaluation costs, highlighting its improved scalability. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_11454 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | UniCBE: An Uniformity-driven Comparing Based Evaluation Framework with Unified Multi-Objective Optimization Yuan, Peiwen Feng, Shaoxiong Li, Yiwei Wang, Xinglin Zhang, Yueqi Shi, Jiayi Tan, Chuyi Pan, Boyuan Hu, Yao Li, Kan Computation and Language Human preference plays a significant role in measuring large language models and guiding them to align with human values. Unfortunately, current comparing-based evaluation (CBE) methods typically focus on a single optimization objective, failing to effectively utilize scarce yet valuable preference signals. To address this, we delve into key factors that can enhance the accuracy, convergence, and scalability of CBE: suppressing sampling bias, balancing descending process of uncertainty, and mitigating updating uncertainty. Following the derived guidelines, we propose UniCBE, a unified uniformity-driven CBE framework which simultaneously optimize these core objectives by constructing and integrating three decoupled sampling probability matrices, each designed to ensure uniformity in specific aspects. We further ablate the optimal tuple sampling and preference aggregation strategies to achieve efficient CBE. On the AlpacaEval benchmark, UniCBE saves over 17% of evaluation budgets while achieving a Pearson correlation with ground truth exceeding 0.995, demonstrating excellent accuracy and convergence. In scenarios where new models are continuously introduced, UniCBE can even save over 50% of evaluation costs, highlighting its improved scalability. |
| title | UniCBE: An Uniformity-driven Comparing Based Evaluation Framework with Unified Multi-Objective Optimization |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2502.11454 |