Balanced Accuracy: The Right Metric for Evaluating LLM Judges -- Explained through Youden's J statistic
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866918405209063424 |
|---|---|
| author | Collot, Stephane Fraser, Colin Zhao, Justin Shen, William F. Willi, Timon Leontiadis, Ilias |
| author_facet | Collot, Stephane Fraser, Colin Zhao, Justin Shen, William F. Willi, Timon Leontiadis, Ilias |
| contents | Rigorous evaluation of large language models (LLMs) relies on comparing models by the prevalence of desirable or undesirable behaviors, such as task pass rates or policy violations. These prevalence estimates are produced by a classifier, either an LLM-as-a-judge or human annotators, making the choice of classifier central to trustworthy evaluation. Common metrics used for this choice, such as Accuracy, Precision, and F1, are sensitive to class imbalance and to arbitrary choices of positive class, and can favor judges that distort prevalence estimates. We show that Youden's $J$ statistic is theoretically aligned with choosing the best judge to compare models, and that Balanced Accuracy is an equivalent linear transformation of $J$. Through both analytical arguments and empirical examples and simulations, we demonstrate how selecting judges using Balanced Accuracy leads to better, more robust classifier selection. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_08121 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Balanced Accuracy: The Right Metric for Evaluating LLM Judges -- Explained through Youden's J statistic Collot, Stephane Fraser, Colin Zhao, Justin Shen, William F. Willi, Timon Leontiadis, Ilias Machine Learning Artificial Intelligence Computation and Language Rigorous evaluation of large language models (LLMs) relies on comparing models by the prevalence of desirable or undesirable behaviors, such as task pass rates or policy violations. These prevalence estimates are produced by a classifier, either an LLM-as-a-judge or human annotators, making the choice of classifier central to trustworthy evaluation. Common metrics used for this choice, such as Accuracy, Precision, and F1, are sensitive to class imbalance and to arbitrary choices of positive class, and can favor judges that distort prevalence estimates. We show that Youden's $J$ statistic is theoretically aligned with choosing the best judge to compare models, and that Balanced Accuracy is an equivalent linear transformation of $J$. Through both analytical arguments and empirical examples and simulations, we demonstrate how selecting judges using Balanced Accuracy leads to better, more robust classifier selection. |
| title | Balanced Accuracy: The Right Metric for Evaluating LLM Judges -- Explained through Youden's J statistic |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2512.08121 |