Copertina

Predicting Disagreement with Human Raters in LLM-as-a-Judge Difficulty Assessment without Using Generation-Time Probability Signals

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autore principale:	Ehara, Yo
Natura:	Preprint
Pubblicazione:	2026
Soggetti:	Computation and Language Computers and Society
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Accurate and Efficient Statistical Testing for Word Semantic Breadth
di: Ehara, Yo
Pubblicazione: (2026)

Can LLMs Estimate Student Struggles? Human-AI Difficulty Alignment with Proficiency Simulation for Item Difficulty Prediction
di: Li, Ming, et al.
Pubblicazione: (2025)

Can LLM be a Personalized Judge?
di: Dong, Yijiang River, et al.
Pubblicazione: (2024)

MLLM-as-a-Judge for Image Safety without Human Labeling
di: Wang, Zhenting, et al.
Pubblicazione: (2024)

Is GPT-4 Alone Sufficient for Automated Essay Scoring?: A Comparative Judgment Approach Based on Rater Cognition
di: Kim, Seungju, et al.
Pubblicazione: (2024)

Vicarious Offense and Noise Audit of Offensive Speech Classifiers: Unifying Human and Machine Disagreement on What is Offensive
di: Weerasooriya, Tharindu Cyril, et al.
Pubblicazione: (2023)

The Gray Area: Characterizing Moderator Disagreement on Reddit
di: Alipour, Shayan, et al.
Pubblicazione: (2026)

Exploring Safety Alignment Evaluation of LLMs in Chinese Mental Health Dialogues via LLM-as-Judge
di: Cai, Yunna, et al.
Pubblicazione: (2025)

Misalignment of LLM-Generated Personas with Human Perceptions in Low-Resource Settings
di: Prama, Tabia Tanzin, et al.
Pubblicazione: (2025)

SMART: Simulated Students Aligned with Item Response Theory for Question Difficulty Prediction
di: Scarlatos, Alexander, et al.
Pubblicazione: (2025)

Controlling Cloze-test Question Item Difficulty with PLM-based Surrogate Models for IRT Assessment
di: Zhang, Jingshen, et al.
Pubblicazione: (2024)

A Cross-Cultural Assessment of Human Ability to Detect LLM-Generated Fake News about South Africa
di: Schlippe, Tim, et al.
Pubblicazione: (2025)

A Scoping Review of LLM-as-a-Judge in Healthcare and the MedJUDGE Framework
di: Li, Chenyu, et al.
Pubblicazione: (2026)

Through the Judge's Eyes: Inferred Thinking Traces Improve Reliability of LLM Raters
di: Zhang, Xingjian, et al.
Pubblicazione: (2025)

A Two-Phase Stability Study of LLM Judges and Bar Council Examiners on Thai Bar-Exam Free-Form Essays
di: Akarajaradwong, Pawitsapak, et al.
Pubblicazione: (2026)

Training-Free Cultural Alignment of Large Language Models via Persona Disagreement
di: Kiet, Huynh Trung, et al.
Pubblicazione: (2026)

Leveraging Explainable AI for LLM Text Attribution: Differentiating Human-Written and Multiple LLMs-Generated Text
di: Najjar, Ayat, et al.
Pubblicazione: (2025)

Estimating Item Difficulty Using Large Language Models and Tree-Based Machine Learning Algorithms
di: Razavi, Pooya, et al.
Pubblicazione: (2025)

Urban Mobility Assessment Using LLMs
di: Bhandari, Prabin, et al.
Pubblicazione: (2024)

Culturally Adaptive Explainable LLM Assessment for Multilingual Information Disorder: A Human-in-the-Loop Approach
di: Jouneghani, Maziar Kianimoghadam
Pubblicazione: (2026)

Explainable Ethical Assessment on Human Behaviors by Generating Conflicting Social Norms
di: Sun, Yuxi, et al.
Pubblicazione: (2025)

Same Verdict, Different Reasons: LLM-as-a-Judge and Clinician Disagreement on Medical Chatbot Completeness
di: DeLucia, Alexandra, et al.
Pubblicazione: (2026)

The Parrot Dilemma: Human-Labeled vs. LLM-augmented Data in Classification Tasks
di: Møller, Anders Giovanni, et al.
Pubblicazione: (2023)

LLM or Human? Perceptions of Trust and Information Quality in Research Summaries
di: Akpinar, Nil-Jana, et al.
Pubblicazione: (2026)

When Can We Trust LLM Graders? Calibrating Confidence for Automated Assessment
di: Ferrer, Robinson, et al.
Pubblicazione: (2026)

Use Me Wisely: AI-Driven Assessment for LLM Prompting Skills Development
di: Ognibene, Dimitri, et al.
Pubblicazione: (2025)

LLM Agents Predict Social Media Reactions but Do Not Outperform Text Classifiers: Benchmarking Simulation Accuracy Using 120K+ Personas of 1511 Humans
di: Bojic, Ljubisa, et al.
Pubblicazione: (2026)

Assessing Judging Bias in Large Reasoning Models: An Empirical Study
di: Wang, Qian, et al.
Pubblicazione: (2025)

Human or LLM as Standardized Patients? A Comparative Study for Medical Education
di: Zhang, Bingquan, et al.
Pubblicazione: (2025)

Leveraging Large Language Models for Predictive Analysis of Human Misery
di: Seal, Bishanka, et al.
Pubblicazione: (2025)

Quantifying and Predicting Disagreement in Graded Human Ratings
di: Zhang, Leixin, et al.
Pubblicazione: (2026)

JudgeMeNot: Personalizing Large Language Models to Emulate Judicial Reasoning in Hebrew
di: Razumenko, Itay, et al.
Pubblicazione: (2026)

Multilingual Prompting for Improving LLM Generation Diversity
di: Wang, Qihan, et al.
Pubblicazione: (2025)

SpiderGen: Towards Procedure Generation For Carbon Life Cycle Assessments with Generative AI
di: Sitaraman, Anupama, et al.
Pubblicazione: (2025)

LLM-Generated or Human-Written? Comparing Review and Non-Review Papers on ArXiv
di: Elazar, Yanai, et al.
Pubblicazione: (2026)

Towards Dynamic Theory of Mind: Evaluating LLM Adaptation to Temporal Evolution of Human States
di: Xiao, Yang, et al.
Pubblicazione: (2025)

The Impact of Item-Writing Flaws on Difficulty and Discrimination in Item Response Theory
di: Schmucker, Robin, et al.
Pubblicazione: (2025)

Augmenting Human-Annotated Training Data with Large Language Model Generation and Distillation in Open-Response Assessment
di: Borchers, Conrad, et al.
Pubblicazione: (2025)

Aligning LLM Uncertainty with Human Disagreement in Subjectivity Analysis
di: Lu, Junyu, et al.
Pubblicazione: (2026)

AI-Augmented Predictions: LLM Assistants Improve Human Forecasting Accuracy
di: Schoenegger, Philipp, et al.
Pubblicazione: (2024)