Beyond Human Judgment: A Bayesian Evaluation of LLMs' Moral Values Understanding
Fuente:
arXiv
Guardado en:
| Autores principales: | , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909915793063936 |
|---|---|
| author | Skorski, Maciej Landowska, Alina |
| author_facet | Skorski, Maciej Landowska, Alina |
| contents | How do Large Language Models understand moral dimensions compared to humans?
This first large-scale Bayesian evaluation of market-leading language models provides the answer. In contrast to prior work using deterministic ground truth (majority or inclusion rules), we model annotator disagreements to capture both aleatoric uncertainty (inherent human disagreement) and epistemic uncertainty (model domain sensitivity). We evaluated the best language models (Claude Sonnet 4, DeepSeek-V3, Llama 4 Maverick) across 250K+ annotations from nearly 700 annotators in 100K+ texts spanning social networks, news and forums.
Our GPU-optimized Bayesian framework processed 1M+ model queries, revealing that AI models typically rank among the top 25\% of human annotators, performing much better than average balanced accuracy. Importantly, we find that AI produces far fewer false negatives than humans, highlighting their more sensitive moral detection capabilities. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_13804 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Beyond Human Judgment: A Bayesian Evaluation of LLMs' Moral Values Understanding Skorski, Maciej Landowska, Alina Computation and Language Human-Computer Interaction 68T50, 62F15, 62P25 I.2.7; K.4.1; J.4 How do Large Language Models understand moral dimensions compared to humans? This first large-scale Bayesian evaluation of market-leading language models provides the answer. In contrast to prior work using deterministic ground truth (majority or inclusion rules), we model annotator disagreements to capture both aleatoric uncertainty (inherent human disagreement) and epistemic uncertainty (model domain sensitivity). We evaluated the best language models (Claude Sonnet 4, DeepSeek-V3, Llama 4 Maverick) across 250K+ annotations from nearly 700 annotators in 100K+ texts spanning social networks, news and forums. Our GPU-optimized Bayesian framework processed 1M+ model queries, revealing that AI models typically rank among the top 25\% of human annotators, performing much better than average balanced accuracy. Importantly, we find that AI produces far fewer false negatives than humans, highlighting their more sensitive moral detection capabilities. |
| title | Beyond Human Judgment: A Bayesian Evaluation of LLMs' Moral Values Understanding |
| topic | Computation and Language Human-Computer Interaction 68T50, 62F15, 62P25 I.2.7; K.4.1; J.4 |
| url | https://arxiv.org/abs/2508.13804 |