Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866914706225102848 |
|---|---|
| author | Chiang, Wei-Lin Zheng, Lianmin Sheng, Ying Angelopoulos, Anastasios Nikolas Li, Tianle Li, Dacheng Zhang, Hao Zhu, Banghua Jordan, Michael Gonzalez, Joseph E. Stoica, Ion |
| author_facet | Chiang, Wei-Lin Zheng, Lianmin Sheng, Ying Angelopoulos, Anastasios Nikolas Li, Tianle Li, Dacheng Zhang, Hao Zhu, Banghua Jordan, Michael Gonzalez, Joseph E. Stoica, Ion |
| contents | Large Language Models (LLMs) have unlocked new capabilities and applications; however, evaluating the alignment with human preferences still poses significant challenges. To address this issue, we introduce Chatbot Arena, an open platform for evaluating LLMs based on human preferences. Our methodology employs a pairwise comparison approach and leverages input from a diverse user base through crowdsourcing. The platform has been operational for several months, amassing over 240K votes. This paper describes the platform, analyzes the data we have collected so far, and explains the tried-and-true statistical methods we are using for efficient and accurate evaluation and ranking of models. We confirm that the crowdsourced questions are sufficiently diverse and discriminating and that the crowdsourced human votes are in good agreement with those of expert raters. These analyses collectively establish a robust foundation for the credibility of Chatbot Arena. Because of its unique value and openness, Chatbot Arena has emerged as one of the most referenced LLM leaderboards, widely cited by leading LLM developers and companies. Our demo is publicly available at \url{https://chat.lmsys.org}. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_04132 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference Chiang, Wei-Lin Zheng, Lianmin Sheng, Ying Angelopoulos, Anastasios Nikolas Li, Tianle Li, Dacheng Zhang, Hao Zhu, Banghua Jordan, Michael Gonzalez, Joseph E. Stoica, Ion Artificial Intelligence Computation and Language Large Language Models (LLMs) have unlocked new capabilities and applications; however, evaluating the alignment with human preferences still poses significant challenges. To address this issue, we introduce Chatbot Arena, an open platform for evaluating LLMs based on human preferences. Our methodology employs a pairwise comparison approach and leverages input from a diverse user base through crowdsourcing. The platform has been operational for several months, amassing over 240K votes. This paper describes the platform, analyzes the data we have collected so far, and explains the tried-and-true statistical methods we are using for efficient and accurate evaluation and ranking of models. We confirm that the crowdsourced questions are sufficiently diverse and discriminating and that the crowdsourced human votes are in good agreement with those of expert raters. These analyses collectively establish a robust foundation for the credibility of Chatbot Arena. Because of its unique value and openness, Chatbot Arena has emerged as one of the most referenced LLM leaderboards, widely cited by leading LLM developers and companies. Our demo is publicly available at \url{https://chat.lmsys.org}. |
| title | Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2403.04132 |