LLM Comparator: Visual Analytics for Side-by-Side Evaluation of Large Language Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866911778296823808 |
|---|---|
| author | Kahng, Minsuk Tenney, Ian Pushkarna, Mahima Liu, Michael Xieyang Wexler, James Reif, Emily Kallarackal, Krystal Chang, Minsuk Terry, Michael Dixon, Lucas |
| author_facet | Kahng, Minsuk Tenney, Ian Pushkarna, Mahima Liu, Michael Xieyang Wexler, James Reif, Emily Kallarackal, Krystal Chang, Minsuk Terry, Michael Dixon, Lucas |
| contents | Automatic side-by-side evaluation has emerged as a promising approach to evaluating the quality of responses from large language models (LLMs). However, analyzing the results from this evaluation approach raises scalability and interpretability challenges. In this paper, we present LLM Comparator, a novel visual analytics tool for interactively analyzing results from automatic side-by-side evaluation. The tool supports interactive workflows for users to understand when and why a model performs better or worse than a baseline model, and how the responses from two models are qualitatively different. We iteratively designed and developed the tool by closely working with researchers and engineers at a large technology company. This paper details the user challenges we identified, the design and development of the tool, and an observational study with participants who regularly evaluate their models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_10524 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LLM Comparator: Visual Analytics for Side-by-Side Evaluation of Large Language Models Kahng, Minsuk Tenney, Ian Pushkarna, Mahima Liu, Michael Xieyang Wexler, James Reif, Emily Kallarackal, Krystal Chang, Minsuk Terry, Michael Dixon, Lucas Human-Computer Interaction Artificial Intelligence Computation and Language Machine Learning Automatic side-by-side evaluation has emerged as a promising approach to evaluating the quality of responses from large language models (LLMs). However, analyzing the results from this evaluation approach raises scalability and interpretability challenges. In this paper, we present LLM Comparator, a novel visual analytics tool for interactively analyzing results from automatic side-by-side evaluation. The tool supports interactive workflows for users to understand when and why a model performs better or worse than a baseline model, and how the responses from two models are qualitatively different. We iteratively designed and developed the tool by closely working with researchers and engineers at a large technology company. This paper details the user challenges we identified, the design and development of the tool, and an observational study with participants who regularly evaluate their models. |
| title | LLM Comparator: Visual Analytics for Side-by-Side Evaluation of Large Language Models |
| topic | Human-Computer Interaction Artificial Intelligence Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2402.10524 |