Imagen de Portada

Is LLM an Overconfident Judge? Unveiling the Capabilities of LLMs in Detecting Offensive Language with Annotation Disagreement

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Lu, Junyu, Ma, Kai, Wang, Kaichun, Xiao, Kelaiti, Lee, Roy Ka-Wei, Xu, Bo, Yang, Liang, Lin, Hongfei
Formato:	Preprint
Publicado:	2025
Materias:	Computation and Language Artificial Intelligence
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

Visual Puns from Idioms: An Iterative LLM-T2IM-MLLM Framework
por: Xiao, Kelaiti, et al.
Publicado: (2025)

Aligning LLM Uncertainty with Human Disagreement in Subjectivity Analysis
por: Lu, Junyu, et al.
Publicado: (2026)

VisualQuest: A Benchmark for Abstract Visual Reasoning in MLLMs
por: Xiao, Kelaiti, et al.
Publicado: (2025)

ToxiCloakCN: Evaluating Robustness of Offensive Language Detection in Chinese with Cloaking Perturbations
por: Xiao, Yunze, et al.
Publicado: (2024)

Multi-Agent VLMs Guided Self-Training with PNU Loss for Low-Resource Offensive Content Detection
por: Wang, Han, et al.
Publicado: (2025)

Towards Patronizing and Condescending Language in Chinese Videos: A Multimodal Dataset and Detector
por: Wang, Hongbo, et al.
Publicado: (2024)

Harder to Defend: Towards Chinese Toxicity Attacks via Implicit Enhancement and Obfuscation Rewriting
por: Kang, Jingyi, et al.
Publicado: (2026)

Overconfidence in LLM-as-a-Judge: Diagnosis and Confidence-Driven Solution
por: Tian, Zailong, et al.
Publicado: (2025)

Vicarious Offense and Noise Audit of Offensive Speech Classifiers: Unifying Human and Machine Disagreement on What is Offensive
por: Weerasooriya, Tharindu Cyril, et al.
Publicado: (2023)

D3CODE: Disentangling Disagreements in Data across Cultures on Offensiveness Detection and Evaluation
por: Davani, Aida Mostafazadeh, et al.
Publicado: (2024)

PclGPT: A Large Language Model for Patronizing and Condescending Language Detection
por: Wang, Hongbo, et al.
Publicado: (2024)

Take its Essence, Discard its Dross! Debiasing for Toxic Language Detection via Counterfactual Causal Effect
por: Lu, Junyu, et al.
Publicado: (2024)

From Text to Emotion: Unveiling the Emotion Annotation Capabilities of LLMs
por: Niu, Minxue, et al.
Publicado: (2024)

When Disagreements Elicit Robustness: Investigating Self-Repair Capabilities under LLM Multi-Agent Disagreements
por: Ju, Tianjie, et al.
Publicado: (2025)

Integrating Multi-view Analysis: Multi-view Mixture-of-Expert for Textual Personality Detection
por: Zhu, Haohao, et al.
Publicado: (2024)

Guardians of Discourse: Evaluating LLMs on Multilingual Offensive Language Detection
por: He, Jianfei, et al.
Publicado: (2024)

Chinese Offensive Language Detection:Current Status and Future Directions
por: Xiao, Yunze, et al.
Publicado: (2024)

Towards Comprehensive Detection of Chinese Harmful Memes
por: Lu, Junyu, et al.
Publicado: (2024)

HateClipSeg: A Segment-Level Annotated Dataset for Fine-Grained Hate Video Detection
por: Wang, Han, et al.
Publicado: (2025)

Language, Culture, and Ideology: Personalizing Offensiveness Detection in Political Tweets with Reasoning LLMs
por: Pihulski, Dzmitry, et al.
Publicado: (2025)

Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge
por: Shi, Lin, et al.
Publicado: (2024)

OCCULT: Evaluating Large Language Models for Offensive Cyber Operation Capabilities
por: Kouremetis, Michael, et al.
Publicado: (2025)

Evaluating Annotation Consistency in Offensive Language Detection: A Data Analytics Approach on the TweetEval Dataset
por: Fabeela Ali Rawther,Abhinay A K,Anagha Tess B,Alan Joseph,Adham Saheer
Publicado: (2025)

With Great Capabilities Come Great Responsibilities: Introducing the Agentic Risk & Capability Framework for Governing Agentic AI Systems
por: Khoo, Shaun, et al.
Publicado: (2025)

The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMs
por: Calderon, Nitay, et al.
Publicado: (2025)

Leveraging Annotator Disagreement for Text Classification
por: Xu, Jin, et al.
Publicado: (2024)

Systematic Capability Benchmarking of Frontier Large Language Models for Offensive Cyber Tasks
por: Merves, Tyler H., et al.
Publicado: (2026)

Do Language Models Mirror Human Confidence? Exploring Psychological Insights to Address Overconfidence in LLMs
por: Xu, Chenjun, et al.
Publicado: (2025)

Taming Overconfidence in LLMs: Reward Calibration in RLHF
por: Leng, Jixuan, et al.
Publicado: (2024)

Towards Effective Offensive Security LLM Agents: Hyperparameter Tuning, LLM as a Judge, and a Lightweight CTF Benchmark
por: Shao, Minghao, et al.
Publicado: (2025)

Heterogeneous Judge-Aware Ranking with Sensitivity, Disagreement, and Confidence
por: Yu, Shibo, et al.
Publicado: (2026)

Function-based Labels for Complementary Recommendation: Definition, Annotation, and LLM-as-a-Judge
por: Yamasaki, Chihiro, et al.
Publicado: (2025)

MemGuard-Alpha: Detecting and Filtering Memorization-Contaminated Signals in LLM-Based Financial Forecasting via Membership Inference and Cross-Model Disagreement
por: Roy, Anisha, et al.
Publicado: (2026)

Same Verdict, Different Reasons: LLM-as-a-Judge and Clinician Disagreement on Medical Chatbot Completeness
por: DeLucia, Alexandra, et al.
Publicado: (2026)

Calibrating Probabilistic Object Detectors with Annotator Disagreement
por: Tan, Zhi Qin, et al.
Publicado: (2026)

Dealing with Annotator Disagreement in Hate Speech Classification
por: Dehghan, Somaiyeh, et al.
Publicado: (2025)

Detection and Analysis of Offensive Online Content in Hausa Language
por: Adam, Fatima Muhammad, et al.
Publicado: (2023)

Offensive Language Detection on Social Media Using XLNet
por: Alothman, Reem, et al.
Publicado: (2025)

Detecting Offensive Memes with Social Biases in Singapore Context Using Multimodal Large Language Models
por: Yuxuan, Cao, et al.
Publicado: (2025)

Can Reasoning Help Large Language Models Capture Human Annotator Disagreement?
por: Ni, Jingwei, et al.
Publicado: (2025)