Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2411.01281 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908615642710016 |
|---|---|
| author | Son, Seonil Oh, Ju-Min Jin, Heegon Jang, Cheolhun Jeong, Jeongbeom Kim, Kuntae |
| author_facet | Son, Seonil Oh, Ju-Min Jin, Heegon Jang, Cheolhun Jeong, Jeongbeom Kim, Kuntae |
| contents | As Large Language Models (LLMs) expand across domains, LLM judges have become essential for systems evaluation. Current benchmarks typically compare system outputs against baselines. This baseline-mediated approach, though convenient, yields lower reliability than direct comparison between systems. We propose Arena-Lite which integrates tournament structure on top of head-to-head comparison. The application of a tournament structure and direct comparison eliminates the need for baseline outputs, reduces the number of required comparisons, and allows higher reliability in system rankings. We conducted two experiments: (1) controlled stochastic modeling and (2) empirical validation with a real LLM judge. Those experiments collectively demonstrate that Arena-Lite consistently achieves higher reliability with fewer comparisons, even with smaller datasets or weaker judges. We release an easy-to-use web demonstration and code to foster adoption of Arena-Lite, streamlining model selection across research and industry communities. Arena-Lite demo and code are available on \href{https://huggingface.co/spaces/NCSOFT/ArenaLite}{https://huggingface.co/spaces/NCSOFT/ArenaLite} |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_01281 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Arena-Lite: Efficient and Reliable Large Language Model Evaluation via Tournament-Based Direct Comparisons Son, Seonil Oh, Ju-Min Jin, Heegon Jang, Cheolhun Jeong, Jeongbeom Kim, Kuntae Computation and Language Artificial Intelligence As Large Language Models (LLMs) expand across domains, LLM judges have become essential for systems evaluation. Current benchmarks typically compare system outputs against baselines. This baseline-mediated approach, though convenient, yields lower reliability than direct comparison between systems. We propose Arena-Lite which integrates tournament structure on top of head-to-head comparison. The application of a tournament structure and direct comparison eliminates the need for baseline outputs, reduces the number of required comparisons, and allows higher reliability in system rankings. We conducted two experiments: (1) controlled stochastic modeling and (2) empirical validation with a real LLM judge. Those experiments collectively demonstrate that Arena-Lite consistently achieves higher reliability with fewer comparisons, even with smaller datasets or weaker judges. We release an easy-to-use web demonstration and code to foster adoption of Arena-Lite, streamlining model selection across research and industry communities. Arena-Lite demo and code are available on \href{https://huggingface.co/spaces/NCSOFT/ArenaLite}{https://huggingface.co/spaces/NCSOFT/ArenaLite} |
| title | Arena-Lite: Efficient and Reliable Large Language Model Evaluation via Tournament-Based Direct Comparisons |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2411.01281 |